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Introduction 



This report summarizes the activities and findings of the external evaluation of the Science: It’s 
Elementary (SIE) program in the period June 2007 through May 2008. In the program’s second 
year, the evaluation collected data from a variety of sources using several types of instruments: 

Questionnaires 

• Administered a questionnaire to collect baseline data on the second cohort of 
participating teachers and schools; 

• Administered a post-professional development questionnaire to gather teachers’ opinions 
of the quality and impact of the professional development; 

• Developed and administered an end-of-year questionnaire to examine the impacts of the 
SIE program on teachers and their teaching; 

• Developed and administered a questionnaire for principals of participating schools; 

• Administered a curriculum survey to Cohort 2 teachers; 

• Developed and administered a curriculum alignment survey for district science 
supervisors/curriculum coordinators ; 

Observations 

• Observed a sample of 36 professional development sessions to examine the quality of the 
training provided to teachers; 

• Conducted 48 classroom observations to examine the fidelity and quality with which 
teachers were implementing the SIE-provided modules; 

Interviews 

• Interviewed a sample of 27 teachers participating in the SIE program; 

• Interviewed a sample of 40 principals (20 from each cohort) whose schools are 
participating in the SIE program; 

• Interviewed a sample of 20 district science supervisors/curriculum coordinators;^ 
Assessments 

• Developed and administered four assessments of teacher content knowledge; 

• Developed seven student assessment scales related to science curriculum modules being 
provided by the program; 

• Administered student assessments for both the Eall 2007 and Spring 2008 module 
implementations in schools participating in the program; and 

• Observed the June 2008 leadership conference designed to prepare teacher leaders to 
provide professional development in future years of the program. 



* For each set of interviews, HRI drew a random initial sample and made repeated attempts to interview everyone in 
the sample. HRI randomly selected additional participants as needed to make up for non-response. In an attempt to 
interview 40 teachers (20 from each cohort), over 100 teachers were contacted via email and telephone. Seventy- 
seven principals and 23 district science supervisors were contacted to reach the targeted goal of 40 and 20 
interviews, respectively. 
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After a brief description of the SIE program and the evaluation plan, this report describes the 
status of the teachers, schools, and districts participating in SIE; the nature and quality of the 
services provided by the program; and preliminary evidence of the program’s impact on teachers, 
their teaching, and their students. The report concludes with a summary of major achievements 
and presents HREs recommendations for the program. 
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Overview oe Science: It's Elementary 



The SIE program is managed by ASSET Inc. and overseen by the Pennsylvania Department of 
Education. SIE is an initiative aimed at improving elementary science instruction across the 
commonwealth of Pennsylvania. The program is focused on helping schools and districts 
implement an inquiry-based, hands-on science education program, with the ultimate goal of 
improving student learning. To accomplish its goals, SIE provides participating schools with 
module -based science instructional materials; teacher professional development around specific 
modules and inquiry -based science teaching more generally; and opportunities for strategic 
planning to help create supportive systems for science education reform. The program has six 
main components: 

1. Strategic Planning Institute: Based on the National Science Resource Center’s EASER 
model, these institutes are intended to help schools and districts understand, plan for, and 
successfully implement inquiry-based elementary science programs. Each school 
participating in SIE sends a team comprised of administrators, teachers, and community 
partners to the Strategic Planning Institute to learn about science education reform and to 
develop a three-year strategic plan for their school. 

2. Vision Conference: This one-day event for school, district, and community leaders is 
intended to help foster a common, long-term vision for science education among a larger 
and more diverse group of stakeholders than the Strategic Planning Institute. Attendees 
include superintendents, district administrators, school board members, principals, and 
community representatives. 

3. Teacher Professional Development: Each teacher responsible for teaching one of the 
SIE-provided science modules is expected to attend an initial three-day training focused 
on that module. This training provides teachers with an opportunity to familiarize 
themselves with the instructional materials; receive practical tips for using them in their 
classrooms; and learn teaching strategies such as inquiry, questioning for higher learning, 
and integrating science and literacy through the use of science notebooks. In the second 
year of the program, teachers are offered a two-day follow-up training on the same 
module to reflect upon their initial implementation of the module and to learn more about 
the related science content. These teachers also begin the professional development cycle 
again for a second module. 

4. Delivery of Classroom Science Materials: In addition to the professional development, 
the SIE program delivers the science modules directly to schools for teachers to use in 
their classrooms. After instruction with the modules is completed, the modules are 
returned and are refurbished for use again the following year. 

5. Eeadership Conference: This five-day conference is intended to help develop the 
capacity of teachers selected by schools participating in SIE to become teacher leaders 
who can mentor and coach other teachers in their schools and districts. The program also 
expects that some of these teacher leaders will be able to serve as module trainers in 
future years. 
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6. Showcase of Student Learning: To help build community support for science education 
improvement, SIE encourages each participating school to host a Showcase of Student 
Learning. During this event, parents, community members, and other stakeholders are 
engaged in science activities that highlight the hands-on, inquiry approach. These 
activities are typically led by students who have experienced this approach in their 
science instruction. SIE provides materials and support to schools for hosting this event. 

Figure 1 shows the program’s theory of action, i.e., how program activities are intended to fit 
together to produce the desired short- and long-term outcomes. 



Science: It’s Elementary Theory of Action 




Figure 1 
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In addition, the SIE program requires each school to designate a Support On Site person (SOS). 
The SOS serves as the main conduit between the SIE program and the school, both for passing 
along important information from the program to the teachers and for helping teachers resolve 
any problems with the modules or professional development that may arise. The SOS is also 
responsible for registering teachers for SIE professional development and coordinating 
evaluation activities for the school. 

In Year Two, SIE added a second cohort of schools to the program. In total, over 2,000 teachers 
in 124 schools participated in the SIE program this year. Table 1 shows the distribution of 
schools across the education regions in Pennsylvania. 



Table 1 

Distribution of Schools Participating in SIE 





Number of Schools 


Region 


Total 


Cohort 1 


Cohort 2 


r 1 


18 


10 


8 


East J 2 


7 


4 


3 


1 3 


7 


4 


3 


r 4 


9 


5 


4 


Central J 5 


16 


9 


7 


1 6 


22 


11 


11 


f 7 


21 


9 


12 


West J 








1 8 


24 


15 


9 


Total 


124 


67 


57 
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Overview of the SIE Evaluation 



The evaluation plan for SIE was developed by HRI in conjunction with key stakeholders with the 
goal of examining the major elements of the program’s theory of action. The questions driving 
the evaluation focus on four main areas: (1) the development of an aligned system to support 
effective science education; (2) professional development for teachers; (3) the impacts of the 
program on teachers and their teaching; (4) and the impacts of the program on students. The key 
evaluation questions, by area, are: 

System Alignment 

1. How effectively is SIE assisting schools in the development of a coherent, cohesive, and 
aligned system to support elementary science education improvement? 

Professional Development 

2. What is the quality of SIE professional development provided to teachers? 

Impacts on Teachers and Their Teaching 

3. What is the impact of SIE on teacher knowledge and .skills? 

4. What is the impact of SIE on teachers’ implementation of the modules? 

5. What is the impact of SIE on student opportunity to learn the Pennsylvania science 
standards? 

Impacts on Students 

6. What is the impact of SIE on student achievement in science? 

This report provides initial answers to these questions based upon the program’s activities in 
Year Two. 
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Supporting a Coherent, Cohesive, and Aligned System 
FOR Elementary Science Education Improvement 



Although enabling schools and districts to create an aligned system is beyond the scope of SIE’s 
mandate, the school and district contexts in which the program is being implemented will likely 
factor into the success of the program. A number of evaluation activities provide data that shed 
light on how these contexts are impacting SIE’s efforts to improve science education. 



Reasons for Participating in SIE 

A questionnaire administered to the principals of schools in the SIE program asked why they 
decided to participate. Principals indicated a desire to improve the science program in their 
schools, in particular making it more inquiry -based, as the main motivator for applying for the 
SIE grant. Nearly three-quarters of the responding principals indicated that the SIE professional 
development and instructional modules were important factors in their decision to join the 
program. (See Table 2.) 



Table 2 

Reasons for Joining SIE 





1" 

Percent of Responding Principals 


Overall 
(N = 93) 


Cohort 1 
(N = 48) 


Cohort 2 
(N = 45) 


Access to the SIE professional development 


74 


74 


73 


Access to the SIE-supported science modules/kits 


72 


79 


64 


Our school’s interest in increasing the amount of inquiry-hased science taught 


71 


83 


58 


The district’s interest in participating 


69 


77 


60 


My (the principal) interest in participating 


68 


65 


71 


Our school’s interest in increasing our teachers’ science content knowledge 


67 


68 


67 


The new science PSSA 


53 


55 


51 


The SIE program’ s alignment with our district’ s vision for its science program 


53 


63 


42 


Our school’s interest in increasing instructional time devoted to science 


48 


56 


40 


Our teachers’ interest in participating 


29 


38 


20 



Includes only those indicating a “4” on a four-point scale of 1 “Not at All” to 4 “To a Great Extent.” 



Interestingly, although only about half of the principals indicated that the new science PSSA 
greatly influenced their decision to join SIE, nearly all of the interviewed district science 
supervisors indicated that it was a factor, though some noted that they had mixed feelings about 
the assessment. As two said: 

The motivating factor, obviously, [is] the accountability with the PSSA assessment in 
science. 

As much as I hate to say it, the science PSSA. 
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Baseline Status of Cohort 2 Schools’ Science Programs 

Overall, the science programs of Cohort 2 schools (i.e., those joining SIE in Year Two) before 
joining SIE were very similar to Cohort 1 schools when they joined the program last year. On a 
district curriculum alignment questionnaire completed by 48 of the 57 Cohort 2 districts, only 69 
percent of respondents indicated that their district had a curriculum guide for science in grades 
K-6. (A copy of the baseline district curriculum alignment questionnaire and frequency 
distributions are included in Appendix A.) In nearly all of these districts, the curriculum guide is 
intended to support teachers in the selection of topics to teach and instructional materials to use; 
however, other data indicate that the intended curriculum is not always being enacted. 

A baseline questionnaire administered to teachers at the beginning of their SIE training, as well 
as interviews with a random sample of principals, indicate that both the emphasis given to, and 
the quality of, prior elementary science instruction was inconsistent, at best. When asked to 
describe the strengths of their school’s science program, one of the most common responses, 
given by nearly one-fifth of responding teachers, was some version of “our school does not have 
a science program.” Often, the lack of a science program was attributed to a heavy focus on 
reading/language arts and mathematics due accountability pressures in those areas. Other 
teachers indicated that, although science is taught, there is no set curriculum for science. 
Principals also acknowledged that the discretion given to teachers often resulted in science being 
deemphasized. Typical responses included: 

Our district focuses on reading and math. The elementary [level] has a limited science 
program. [Teacher] 

Our program currently has no strengths. It is inconsistent. We have no materials and no 
time allotted for science. [Teacher] 

Well basically our science program was in poor condition. Our curriculum was not 
aligned to the standards. We had focused mainly on reading and math and writing. The 
reason why we got in Science: It’s Elementary is because we were just starting to look at 
our science curriculum which needs to be revised. [Principal] 

These kinds of concerns were echoed by district science supervisors. In interviews, many 
science supervisors stated that they thought teachers were spending the vast majority of their 
instructional time on mathematics and reading due to the high-stakes nature of the PSSA in those 
subjects. 

The relative lack of emphasis on science instruction prior to SIE participation is also evident 
from teacher responses to close-ended items on the questionnaire. As can be seen in Table 3, 
fewer than half of the teachers indicated teaching science four or five times per week. The 
length of science lessons varied greatly, with most lasting between 21 and 50 minutes. Taken 
together, these responses translate to an average of 22 minutes of science instruction per day. 
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Table 3 

Time Devoted to Science Instruction Prior to SIE in Cohort 2 Schools 





Percent of Teachers 
(N = 1,026) 


Science Lessons per Week 




0 


7 


1 


11 


2 


14 


3 


26 


4 


17 


5 


25 


Minutes per Lesson 




10 or fewer 


5 


11-20 


9 


21-30 


29 


31-40 


29 


41-50 


20 


5 1 or more 


8 



Interview responses noting strengths of pre-SIE science programs varied greatly and few strong 
themes emerged. About 1 in 3 teachers thought that their existing instructional materials were of 
high quality; roughly half of these teachers had previously used kits like those provided by SIE. 
Other responses, each given by a handful of teachers, included the quality of district support 
staff, and teachers’ positive attitudes toward teaching science. 

When asked in what ways their school science program could be improved, teachers typically 
focused on allocating more time for science teaching, access to materials for teaching science, 
and/or increased use of hands-on science instruction. Typical responses included: 

We would need to lengthen the day in order to teach an appropriate science program. 

The reading and math programs that we follow take up at least 90 percent of our 
teaching day. 

[A] stronger thematic approach developed from pre-K up. We should work together to 
build on concepts and themes, and be able to do more experimentation. 

In order to help examine curriculum alignment with the state standards, and to provide data that 
could assist schools, districts, and the Pennsylvania Department of Education (PDE) with 
curriculum planning, a teacher curriculum survey was administered to all teachers in the Cohort 
2 SIE schools responsible for teaching science. The survey was modeled after the Surveys of 
Enacted Curriculum,^ asking teachers how much instructional time was devoted to the various 
Pennsylvania science standards and the nature of that instruction. 



^ The questionnaire was conducted on-line, with each school’s Support on Site person (SOS) responsible for 
administering it to the teachers at the school. Of the 1,027 teachers registered for the survey by their SOS, 873 
completed the questionnaire, a response rate of 85 percent. 

^ Blank, R. K., Porter, A., & Smithson, J. (2001). New tools for analyzing teaching, curriculum and standards in 
mathematics and science: Results from survey of enacted curriculum project, final report. Washington, DC; Council 
of Chief State School Officers. 
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Table 4 shows the amount of instructional time teachers reported devoting to the Science and 
Technology and Environment and Ecology standards, disaggregated by grade level expectations. 
Overall, about two-thirds of instructional time is reportedly spent on Science and Technology 
standards, and one-third on Environment and Ecology standards. 



Table 4 

Instructional Time Devoted to Categories of Standards in Cohort 2 Schools, 
by Grade Range 





Percent of Instructional Time 


All Teachers 
(N = 873) 


K-4 Teachers 
(N = 728) 


5-6 Teachers 
(N = 145) 


4'*’ Grade Science and Technology 


35 


35 


35 


4* Grade Environment and Ecology 


21 


21 


16 


7* Grade Science and Technology 


30 


29 


36 


7* Grade Environment and Ecology 


15 


15 


13 



It is interesting to note that grades K-4 teachers reported spending 44 percent of their 
instructional time on standards that the state designates as those to be addressed in grades 5-7; 
grades 5-6 teachers reported devoting over 50 percent of their science instruction to 4* grade 
standards. 

Although one would expect some instructional time to be “off grade level” (e.g., for 
reinforcement, extensions, and making connections), these data seem to indicate that at this point 
the enacted curricula in many of the schools participating in the SIE program are not well aligned 
with the state standards. Additional data from this survey, disaggregated by major topic area 
(e.g.. Inquiry and Design; Biological Sciences, Earth Sciences), are presented in Appendix B. 

It will be important for schools, districts, and PDE to consider how well the data from this survey 
match their vision for science curriculum, and what steps, if any, should be taken to improve 
alignment.^ In addition, it is HRI’s experience that most teachers do not have the time or 
training to develop high-quality curriculum materials. Thus, the SIE program may want to 
consider ways to assist schools and districts in developing curricula that are well aligned with the 
state science standards. 



To facilitate such work, HRI has made each SIE school’s data available to school personnel via the on-line system 
created for the curriculum survey. In addition, PDE has access to all schools’ data, individually and in aggregate, 
via the on-line system. 
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Status of Cohort 1 Schools’ Science Programs 

Although Cohort 2 schools were just beginning the process of improving their science programs, 
Cohort 1 schools were in their second year of the process. In interviews, a sample of 20 
principals of Cohort 1 schools was asked about changes that have occurred in their schools since 
joining SIE. Several of the interviewed principals indicated that their science program had 
changed to a more hands-on, inquiry-based approach to science. A typical response was: 

Before, there was a lot of reading, recall from textbooks. Now it’s a lot of hands-on. 

In addition, most of the principals indicated that the amount of instructional time devoted to 
science had increased. Some principals indicated that the school now has a scheduled time for 
science. Others indicated that the school had lengthened the time schedule for science. As one 
principal said: 

Before, science was one of the subjects that if we ran out of time, it would be cut, you 
know, so that they could do more with reading or math. And now, science has a 
scheduled time. 

Although most principals thought the increased emphasis on science was a positive outcome, a 
few expressed concern about the increased time, particularly in kindergarten and the early 
elementary grades. As one principal said when asked if there had been changes to the amount of 
time devoted to science: 

Oh, definitely. They [teachers] have no choice but to spend more time on it, which is 
very stressful. ..especially in the primary grades. The primary grades are really 
struggling with the amount of time it takes to do the modules. And we have 
kindergarteners here for just a tad over two hours. And they have a half an hour 
expressive arts time in that block. So when do you do all of your literacy, your math and 
now in addition aforty-five minute science? 

Principals were also asked if there had been changes in the science topics taught in their schools. 
Several principals noted that a greater number of science topics were being taught. Others 
indicated that science is now covered at greater depth and that their science curriculum is better 
aligned with state standards. Principals generally attributed these changes to the implementation 
of the SIE-provided modules, and indicated that the SIE professional development enabled these 
changes to occur. As one principal commented: 

7 have never had 20 teachers go out to a training, and every single one of them came 
back with glowing remarks about the training, about how excited they were to have that 
training. 

Einally, many of the principals noted a renewed excitement for science in their students, and that 
this enthusiasm has even gotten parents excited about science. As three principals said: 
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You know the kids love it. I think it makes a difference for everybody. The kids enjoy it, 
they talk about it, they ’re actually really engaged in it and you know that makes it worth 
it to everybody. 

The kids are excited about science. They get excited when they know they ’re starting the 
modules. 

And I think the children are going home and even amazing their parents. [A child’s] 
mother says to me, “Oh, my goodness, the things he comes home and talks about and can 
really explain it. ’’ 

Three-quarters of the interviewed principals indicated that one of their teachers had been trained 
as an SIE teacher leader in last year’s leadership conference; most of these indicated that this 
teacher leader acted as a resource person for other teachers in their school, helping other teachers 
with the implementation of the modules; about half reported that the teacher leaders had taken on 
other responsibilities in the school as well such as serving on the science fair committee, acting 
as the SOS, and answering parents’ questions about SIE. Comments about the teacher leaders 
included: 

She’s looked at definitely in my building as much more of a resource person, she's able to 
answer a lot of the questions as the teachers are going through the kits. 

[The teacher leader is seen as a] mentor, a facilitator, someone they can go to with 
questions; people view her as the expert. 

The sole drawback to the teacher leader portion of the program identified by the principals was 
the amount of time the teacher leader was taken out of the classroom. As one principal said: 

They’re out of the class for the training more. And that’s the killer in this whole thing. 



Factors Affecting Science Program Improvement 

On a questionnaire, principals of both cohorts were asked about factors that facilitate or inhibit 
their school’s science program. As can be seen in Table 5, nearly all principals indicated that 
access to professional development for teachers as well as the quality and availability of 
resources for science instruction (presumably those being provided by SIE) were facilitators of 
science instruction. State and district science curriculum frameworks were also frequently seen 
as helpful. State and district testing policies in subjects other than science were the most 
frequently mentioned inhibitors, though these were selected by only 1 in 5, or fewer, principals. 
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Table 5 



Principal Opinions of Factors that Affect Science Instruction 





Percent of Responding Principals 
(N = 93) 


Facilitates^ 


Inhibits* 


Access to professional development for science 


92 


7 


Quality of resources for science instruction 


90 


5 


Availability of resources for science instruction 


89 


9 


District science curriculum frameworks 


85 


3 


State science curriculum frameworks 


85 


5 


State science testing policies and practices 


81 


10 


Importance that the district places on science instruction 


77 


3 


Consistency of science reform efforts with other district/school reforms 


70 


4 


State testing policies and practices in subjects other than science 


62 


20 


District science testing policies and practices 


60 


3 


District testing policies and practices in subjects other than science 


57 


15 


District grading policies and practices 


47 


8 


District policies for evaluating teachers 


39 


2 


District structures for recognizing and rewarding teachers 


24 


2 



Includes those responding “Somewhat Facilitates” or “Greatly Facilitates.” 
* Includes those responding “Somewhat Inhibits” or “Greatly Inhibits.” 



Principals were also asked what additional services the SIE program could provide that would 
help them support their school’s science program. Ideas for securing resources for science 
instruction; creating opportunities for teacher collaboration; assessing progress of the school’s 
science program; integrating science with mathematics and literacy; and providing feedback to 
teachers were each mentioned by roughly 60 percent of principals. (See Table 6.) 



Table 6 



Principals Indicating that Potential Program Services Would be Very Valuable 





Percent of Responding Principals 


Overall 
(N = 93) 


Cohort 1 
(N = 48) 


Cohort 2 
(N = 45) 


Ideas and tools for securing the resources needed for science instruction 


63 


57 


70 


Ideas and tools for creating opportunities for teacher collaboration 


62 


56 


69 


Ideas and tools for assessing progress of our school’s science program 


61 


63 


60 


Professional development for me to increase my own understanding of how to 








integrate science with mathematics and literacy 


60 


56 


64 


Ideas and tools for observing and providing feedback to teachers about their 








science instruction 


59 


56 


62 


Ideas and tools for increasing parental support/interest/involvement in our 








school’s science program 


46 


48 


44 


Professional development for me to increase my own understanding of effective 








science teaching 


43 


46 


40 


Ideas and tools for increasing community support/interest/involvement in our 








school’s science program 


42 


42 


43 


Ideas and tools for leveraging other district resources (e.g., professional 








development days) to support science instruction 


42 


33 


51 


Ideas and tools for making science more of a priority in the school 


39 


33 


44 


Professional development for me to increase my own science content 








knowledge 


25 


29 


20 
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The Strategic Planning Institute and Vision Conference 

The Strategic Planning Institute (SPI), and to a lesser extent the Vision Conference, are SIE’s 
main activities for helping schools new to the program develop a common vision for science 
instruction, as well as a plan for making that vision a reality. The SPI is a five-day event 
modeled after the National Science Resource Center’s LASER Institute. Each SIE school was 
invited to send a five-person team to the SPI, ideally a mix of district administrators, school 
administrators, teachers, and community representatives. The SPI had six main goals: 

1. Develop a shared vision about effective science teaching and learning; 

2. Design models for developing and sustaining a corps of leaders for improving the 
teaching and learning of science; 

3. Apply knowledge about the five components of an effective school-based infrastructure 
needed to support a research-based science program (i.e., standards-based materials, 
professional development, centralized materials support, assessment, and 
community /administrative involvement) ; 

4. Develop a strategic plan for improving instruction in science that is informed by research, 
incorporates best practices, and leverages change through partnerships; 

5. Assess and incorporate local, state, regional, and national resources that will contribute to 
the implementation and sustainability of a research-based science program; and 

6. Become members of a network of informed professionals working to increase student 
achievement through the implementation of effective science programs. 

To help accomplish these goals, the SPI offered a number of sessions that focused on topics such 
as systemic change, research about science teaching and learning, the LOSS and STC modules 
that are part of the SIE program, and research about effective professional development. The 
SPI also provided time for teams to generate a strategic plan for improving science education in 
their schools and districts. 

Interviews with principals, teachers, and district science supervisors indicate that attendees 
benefited from the SPI. Many of the interviewees reported that it was helpful to have time to sit 
down as a team and think about their science program, including looking at both what was taught 
in their schools and how it was taught, as was having the opportunity to begin developing a plan 
for improving their science program. 

The Strategic Planning Institute was a real eye-opening experience for us because it gave 
us the opportunity to sit down and make some long-term decisions about where science is 
going. It was a good week. It was very informative and lots of good networking was 
done, not only amongst the people that we brought from our district, but with other 
districts. It provided a lot of information that I brought back to the board and it really 
helped build the momentum for this project. [District Science Supervisor] 
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[We got] a better layout, a better feel for what was going to really be needed to 
implement this and make sure that our teachers bought into it... [The SPI[ laid the 
groundwork for successful implementation. [Principal] 

It gave us a great oversight of the entire program. . .It also allowed us to look at the 
different modules, trying to decide where our weakness was as a school and the content 
that we needed to pick to strengthen our curriculum, and by looking at those modules that 
were on display helped a great deal. We were able to pick the module that we felt best 
suited our curriculum at each grade level. We came back with a good plan. ..good vision 
that we could explain to our community and our fellow teachers, what this Science: It’s 
Elementary is all about. [Teacher] 

The major concern about the SPI, arising from all three sets of interviews, was that it required 
too great a time commitment. A number of the attendees also indicated that the SPI devoted too 
much time to trying to convince them of the value of the program, which they thought was 
unnecessary given they had applied to be part of SIE. As one district science supervisor said: 

We heard a lot of things that were being said, or presented, to sell us on inquiry-based 
instruction. We didn ’t need to hear that because we already know that that’s what we 
wanted to do, and knew the importance, etc. 

It would be advisable for SIE to re-examine the agenda of the SPI to determine whether and how 
elements can be condensed or modified to be more intentional in helping participants develop a 
common vision of effective science instruction, in particular downplaying efforts to “sell” them 
on the program. 

The Vision Conference is a one-day event designed to help build support for an improved 
science education program among a larger and more diverse group of stakeholders than the SPI 
participants. Attendees included superintendents, district administrators, school board members, 
community representatives, and principals of participating schools. According to interviewees, 
the Vision Conference was effective at building a broader base of support for the program. The 
following response was typical: 

Just seeing what the vision was for the program [was helpful]. You know, the beliefs in 
the teaching methods that are being used, how to get the community more involved in the 
program. . . [Principal] 

The most commonly mentioned suggestion for improving the Vision Conference centered on its 
redundancy with the SPI. Although the Vision Conference is aimed at a broader audience than 
the SPI, many schools sent the same people to both events. Thus, the program may want to 
consider clarifying expectations as to who should attend which events and make sure to 
disseminate those expectations as early as possible. Another possibility would be to fold the 
Vision Conference into the SPI, inviting school board and community members to a day of the 
SPI that focused heavily on developing a vision for effective science instruction. 
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The Quality of SIE Professional Development 



The primary aim of the SIE program is to enhance student learning of science in K-6 classrooms 
by providing high quality instructional materials and professional development to help teachers 
implement those materials well. The consensus view among experts in the field is that 
professional development needs to be sustained over time, situated in teachers’ classroom 
practice, and focused on issues of curriculum and pedagogy related to the learning of specific 
content.^ Professional development should also model and then explicitly discuss effective 
instruction and involve multiple stakeholders within schools. 

The design of the SIE professional development program incorporates this wisdom of practice by 
providing teachers with a sequence of opportunities to move from mechanical (novice use) to 
highly purposeful (expert use) implementers of the instructional materials. Error! Reference 
source not found, summarizes the knowledge and skills HRI thinks teachers need to 
purposefully implement a module. 



^ See for example: Elmore, R. F. (2002). Bridging the gap between standards and achievement: The imperative for 
professional development in education. Washington, DC: Albert Shanker Institute. 

Banilower, E. R., Boyd, S. E., Pasley, J. D., & Weiss, I. R. (2006). Lessons from a decade of mathematics and 
science reform: A capstone report for the local systemic change through teacher enhancement initiative. Chapel 
Hill, NC: Horizon Research, Inc. 
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Knowledge and Skills Needed for Purposeful Implementation of a Module 
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First, teachers need to know and feel comfortable with “what’s in the box,” i.e., the management 
of materials and students while using a module, including what the activities are in a module, 
how to set-up and use the provided equipment, and how students are supposed to interact with 
the materials. 

Second, in order to effectively teach the science content to students, teachers must understand the 
content themselves. At a minimum, teachers should understand the ideas students are expected 
to learn, as well as how those ideas fit together in a coherent and cohesive framework (i.e., 
interrelationships among the ideas rather than isolated concepts or factoids) and build toward the 
“big ideas” of science. Teachers would also benefit by understanding the content beyond the 
student level, including how the ideas progress through the K-12 sequence, so they can guide 
students along productive paths of inquiry. 

Third, in order to teach effectively, teachers need to have a clear understanding of how students 
learn science and an explicit vision of effective instruction. A great deal has been learned in the 
last few years about how people learn and the implications of research in the cognitive sciences 
for instruction.^ 

For instruction to be purposeful and effective, teachers need to be able to integrate their 
knowledge of the science content and their familiarity with the nuts and bolts of the module, 
understanding the “content storyline,” how each of the scientific ideas addressed by the module 
is developed through specific learning experiences in the module. Ideally, teachers would be 
able to trace the development of a scientific idea through a module (e.g., students’ initial ideas 
about a particular idea are elicited in activity 1; the idea is then developed in activity 2; students 
have an opportunity to practice and master the idea in activity 3). 

Similarly, for instruction to be purposeful and effective, teachers need to be able to integrate 
their knowledge of the science content with their understanding of how students learn science. 
This type of knowledge is commonly referred to as pedagogical content knowledge and includes 
what initial ideas students are likely to have that may get in the way of their learning the targeted 
concepts (often termed “misconceptions” or “naive ideas”), and specific strategies for helping 
students develop the correct ideas. For example, students often think that connecting a light bulb 
to one end of a battery will cause the bulb to light. A specific strategy for helping students move 
past this idea is to have them try many different configurations of connecting the bulb to the 
battery, recording which do and do not work, and then carefully analyzing the results to find the 
commonalities in the configurations that work, and the commonalities in those that do not work. 

Finally, it is important to note that general pedagogical and classroom management skills are 
necessary for effective teaching of any topic. The SIE program is also attempting to broaden 



® National Research Council. (2000). How people learn: Brain, mind, experience, and school. J. D. Bransford, A. L. 
Brown, & R. R. Cocking (Eds.). Washington, DC: National Academy Press. 

National Research Council. (2005). How students learn: Science in the classroom. M. S. Donovan & J. D. 
Bransford, (Eds.) Washington, DC: National Academy Press. 
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teachers’ “tool bag” of skills, including ones that allow for the integration of reading/language 
arts instruction with the teaching of science (e.g., notebooking strategies). 

In Year Two, SIE provided two types of professional development to address the needs of 
classroom teachers and work toward purposeful use of the modules. Initial Module Training was 
provided to teachers prior to their first implementation of a module. For teachers new to SIE, 
this training spans three days. The first day centers on orienting teachers to SIE, inquiry -based 
science teaching, and using science notebooks. The next two days focus on the SIE-provided 
module. Teachers being trained on additional modules attend only the latter two days of the 
Initial Module Training. The Initial Module Trainings are intended to help teachers attain at 
least a mechanical use of the module, and to lay the groundwork for further professional 
development that will move them toward more purposeful use. 

SIE also offered two-day Module Enrichment Workshops in Year Two. Module Enrichment 
Workshops are intended to deepen the science and pedagogical content knowledge of teachers 
around the topics in a module. These workshops are aimed at teachers that have already attended 

n 

the initial training and have implemented the module in their classrooms. Most of the teachers 
attending the enrichment workshops in Year Two did so after attending the initial training on a 
second module. Table 7 shows the number of teachers completing the post-professional 
development teacher questionnaire at each type of workshop. 



Table 7 



Responses to Post-Professional Development Questionnaire 





Number of Teachers 


Cohort 1, module enrichment on module 


590 


Cohort 1, initial training on Z"** module 


1,025 


Cohort 2, initial training on T* module 


1,093 



Initial Module Trainings 

As noted above, the goal of the Initial Module Training workshops is to prepare teachers to use 
the instructional materials, including familiarizing them with the activities, the content taught in 
the activities, how the content is developed in the instructional materials, and strategies for 
engaging students with the content. Data for this section of the report come from questionnaires 
completed by teachers attending these trainings, interviews with a sample of 37 participating 
teachers, interviews with a sample of 40 principals, and observations of a sample of 26 
workshops. 



7 

The program has decided that teachers need more experience using the modules before they are ready to delve 
deeper into the content of the modules; thus, starting in 2008-09, Module Enrichment Workshops will be offered to 
teachers only after they have implemented a module twice in their classroom. 
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Teachers’ Needs and Expectations 

Knowing what participants’ needs and expectations are for a professional development 
experience can provide valuable insight for understanding their perceptions of the quality of that 
experience. An open-ended item on the baseline questionnaire asked Cohort 2 teachers what 
they hoped to gain from participating in SIE professional development. The most common 
response, given by slightly less than half of Cohort 2 teachers, was that they were hoping to learn 
new instructional strategies. Teachers specifically mentioned learning more student-directed, 
hands-on, and inquiry techniques. Typical responses included: 

New ideas to help students retain the knowledge they are learning in the hands 
on activities. 

Teaching strategies to be more ejfective, instructional practices in all areas of 
content, not just science. 

One-quarter of the participants indicated their primary concern was learning how to use the 
modules being provided by SIE. Eor example: 

Just an idea [ of] what the kit contains and what we ’re going to be teaching. How to 
implement the activities. 

I want to know what is involved in using the kits and how SIE will positively 
impact my students. I hope to gain knowledge on how to successfully use the kits. 

A handful of teachers also indicated that they were hoping to learn how to make science more 
enjoyable for their students. As two participants wrote: 

I hope to learn activities that will help students understand science while making 
it fun and exciting. 

I am hoping to receive a module that is easy to use, interesting to my students, 
and will not require a lot of additional preparation time on my part. 

Data from teacher interviews coincide with the questionnaire data with one exception. A number 
of interviewed participants indicated that they were hoping for help on how to manage time 
effectively when using the materials. As one interviewee said when asked what expectations 
s/he had for the workshops: 

How to put this system into the time that I have allotted to science. ..I don ’t have a whole 
lot of time for science, but I have to cover the same amount. ..do all these experiments, 
and... all the preparation. 

Teachers’ Perceptions of the Quality of the Initial Module Trainings 
Overall, teachers had very positive opinions about the SIE professional development. A 
questionnaire administered at the end of each session asked teachers about the quality of the 
training. As can be seen in Table 8, nearly all teachers thought the facilitators were well 
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prepared and encouraged active participation, and that they provided useful tips for successfully 
implementing the modules. In addition, the vast majority of teachers indicated that the Initial 
Module Training sessions increased their understanding of the science content and their 
familiarity with the activities in the modules. 



Table 8 

Teacher Opinions of the Quality of SIE Initial Module Training 





Percent Agreeing^ 




Cohort 1, 


Cohort 2, 




2nd Module 


1st Module 




(N = 1,093) 


(N = 1,025) 


The facilitators shared tips and suggestions for successfully implementing the module 
The professional development increased my understanding of the science content in 


95 


98 


the module 


95 


95 


The professional development familiarized me with the activities and materials in the 






module 


95 


97 


The facilitators encouraged active participation and investigation by all participants 


94 


97 


The goals of the professional development were clear 


94 


96 


The professional development prepared me to use the teaching strategies promoted by 






SIE 


94 


95 


The facilitators were well prepared 


93 


95 


The professional development reflected careful planning and organization 
Question about the science content in the module were adequately addressed by the 


93 


95 


facilitators 


93 


94 


The various components of the training were useful in meeting the goals of this 






professional development 


93 


96 


The facilitators modeled effective teaching strategies 

The facilitators explicitly discussed how, when and why to use different teaching 


92 


94 


strategies 

The professional development increased my confidence in my ability to teach using the 


92 


93 


SIE module 


92 


91 


The professional development was relevant to my classroom instruction 

There were opportunities for participants to express their vies and collaborate with 


92 


90 


peers 


91 


93 


The professional development was worth the time that I invested 


90 


91 


I would recommend this professional development to a colleague 

Adequate time, structure, and guidance were provided for participants to reflect 


90 


90 


individually on the substance of the professional development 
Adequate, time, structure, and guidance were provided for participants to discuss the 


87 


89 


SIE modules and pedagogical strategies with each other 


86 


87 



Includes those teachers agreeing or strongly agreeing with each statement. 



Participants were also asked about the pace of the professional development. Roughly three- 
quarters of the teachers indicated that the pace was appropriate. (See Table 9.) Twenty-one 
percent of Cohort 2 teachers and 13 percent of Cohort 1 teachers found the pace of the Initial 
Module Trainings too slow; 10 percent of teachers indicated the pace was too fast. 
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Table 9 



Teacher Opinions of the Pace of SIE Initial Module Trainings 





Percent of Teachers 


Cohort 1, 
2”“ Module 
(N = 1,059) 


Cohort 2, 
1®‘ Module 
(N = 996) 


Much too slow 


2 


3 


Somewhat too slow 


12 


18 


Appropriate 


77 


69 


Somewhat too fast 


9 


9 


Much too fast 


1 


1 



When asked in an open-ended question what aspects of the professional development were most 
helpful or effective, two-thirds of the responding teachers indicated going through the module 
and doing the activities as students. Quite a few respondents also mentioned the helpfulness of 
the facilitators sharing implementation strategies, and many described the collaboration with 
their colleagues. In interviews, both teachers and principals provided positive feedback about 
the professional development. Typical responses included: 

I thought it was good that they had us actually doing the experiments and the 
observations that the children would be doing. And actually make the notebook so we 
could take that back with us and see what that would be like. Practicing new strategies 
that they presented to have the children share what they learned. Because it was hands- 
on, it was good for getting us ready to do it with the children. [Teacher] 

[It was helpful that the PD was provided] by teachers who had already taught the 
modules with students. I think that was good because they could tell us where the 
problems might be, or different ways of doing it to make our teaching easier with the 
different investigations. It was actually teachers who had gone through the modules 
themselves with students. [Teacher] 

The professional development really seemed to fit the needs of the teachers, giving them 
the content knowledge they needed, giving them the motivation they needed, and the 
instructional strategies they would need to implement it. [Principal] 

When interviewed teachers were asked how the professional development could be improved, 
their most common response was that it didn’t need any changes, another indicator of their 
satisfaction with the workshops. Still, a number of teachers did have suggestions for 
improvement that are worth considering. Some participants indicated that the orientation to 
inquiry-based instruction on the first day of the three-day training wasn’t as helpful as the rest of 
the training. For example: 

7 have to say that in the beginning, there was just a lot of preliminary stuff we probably 
could have done without, you know all that role playing and talking and all of that. 
Maybe cutting out some of that... some of the fluff and getting right to why you are there. 



Horizon Research, Inc. 



22 



October 2008 




Some teachers suggested that more time needed to be spent on the module and how they could 
implement it in their classrooms. As some of these teachers said: 

~We went through the beginning ones extensively, and when I was teaching them, I felt 
more comfortable with what was happening, but being that we sped through the last ones, 
I am a little bit apprehensive going into it. I feel like I need to prepare more. 

As far as how the training could’ve been made more useful, if they would’ve been able to 
address some issues about how to fit this in to various schedules, time management kind 
of things, that would be good. 

The end-of-year questionnaire also asked teachers what, if anything, the program could do to be 
more effective. The most common response among both Cohort 1 and Cohort 2 teachers 
centered on the logistics of the training sessions. Not surprisingly, teachers indicated they would 
prefer the trainings to be offered at locations closer to their homes and that the required training 
be shortened so they would be out of their classrooms fewer days. Teachers also indicated that 
the trainings would be more helpful if they occurred nearer to the time of module 
implementation. With the number of teachers served by the program and the costs associated 
with offering a training session, the program cannot offer trainings at locations and times ideal 
for every participant. However, given how busy teachers are and how difficult it is for them to 
be out of the classroom, the program may want to consider offering some trainings in the 
summer or on weekends. 

Observers’ Perceptions of the Quality of the Initial Module Trainings 
Preparing teachers to implement a new instructional program is a major undertaking and a new 
science instructional program is particularly challenging. Many teachers (in Pennsylvania and 
across the nation) have put science instruction on the back burner, both because of relatively 
weak content backgrounds in science and high-stakes testing in mathematics and 
reading/language arts. 

SIE’s plan took into account research on teacher concerns suggesting that teachers using new 
instructional materials will first be concerned with the mechanics of the activities and their role 
in leading them, so initial training will necessarily be focused largely on familiarizing teachers 

o 

with the basics of the module. SIE’s goals for the Initial Module Trainings were: 

1. Introduce teachers to Science: It’s Elementary (at the introductory day only); 

2. Prepare teachers to implement the module at the mechanical level; 

3. Deepen teachers’ understanding of the content in the module at the student level; 

4. Deepen teachers’ understanding of the content storyline of the module; and 

5. Prepare teachers to use instructional strategies, specifically notebooking and questioning, 
that support module implementation. 



* Hall, G. & Loucks, S. (1979). Implementing innovations in schools: A concerns-based approach. Austin, TX: 
Research and Development Center for Teacher Education, University of Texas. 
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Trained researchers observed 26 Initial Module Training sessions between October 2007 and 
January 2008. Table 10 shows the number of sessions observed for each module. 



Table 10 



Initial Module Trainings Observed 





Number of Sessions 


Chemical Tests 


3 


Ecosystems 


2 


Electric Circuits 


3 


Environments 


4 


Human Body 


4 


Levers and Pulleys 


3 


Magnets and Motors 


1 


Mixtures and Solutions 


1 


Motion and Design 


2 


Plant Growth and Development 


1 


Rocks and Minerals 


2 



To examine the extent to which the SIE professional development provided teachers with 
opportunity to achieve the learning goals, HRI utilized a framework based on research on how 
people learn. ^ The framework specifies that effective science professional development provides 
accurate information about disciplinary content and the nature of science; enables participants to 
engage intellectually with important ideas; provides opportunities for participants to reflect on 
and make sense of the key ideas of the session; and provides opportunities for participants to 
relate the ideas in the session to their work as classroom teachers; all of which is facilitated by a 
collaborative culture. An analysis of SIE professional development by each of the program’s 
specified goals follows. 

• Introducing teachers to Science: It’s Elementary 
The initial day of training for Cohort 2 teachers focused on familiarizing them with SIE, module- 
based science instruction, inquiry, and literacy connections to science. Answering teachers’ 
questions about SIE early on was especially important so that they could engage with the main 
goals of the module training. An icebreaker activity was used to identify participants’ burning 
questions about SIE. Participants were asked to describe on pieces of chart paper around the 
room their current understanding, and any questions they had, about inquiry, science notebooks, 
science education reform, and the SIE program. This activity allowed facilitators to see where 
the group stood in their understanding of these topics, and to tailor the information they provided 
throughout the session to the information needs of the particular group. 

Eacilitators then provided an overview of the SIE program that explained the elements of science 
education reform as envisioned by the National Science Resources Center, and how the SIE 
program addresses these elements. Teachers had opportunities to ask questions about the 
program throughout the training, as facilitators utilized the “parking lot” strategy (teachers could 



^ HRI’s frame for thinking about the quality of instruction for both adult and student learners is based upon the 
National Research Council’s How people learn: Brain, mind, experience, and school (1999) and How students 
learn: Science in the classroom (2005). 



Horizon Research, Inc. 



24 



October 2008 




post questions on chart paper that would be discussed when time allowed). These strategies for 
dealing with teachers’ initial information needs appear to have been successful, allowing 
teachers to focus on the module- specific aspects of the training. 

As part of the baseline and post-professional development questionnaires, HRI administered a set 
of items developed as part of the Concerns Based Adoption Model (CBAM). The theory holds 
that when teachers adopt a new curriculum or instructional approach, they progress through a 
series of concerns about the “innovation.” As experience with the innovation grows, early 
concerns (e.g., being unfamiliar with the curriculum) are resolved, and later ones emerge. The 
implication of this stage theory is that early concerns must be resolved before later ones can be 
effectively addressed. The CBAM theory suggests that effective professional development 
specifically targets the concerns that are likely to emerge at different points in the adoption 
process. The seven stages of concern are operationalized in a 35-item CBAM questionnaire^*^ 
developed to measure the intensity of each concern: 

1. Awareness (e.g., I don’t even know what the SIE modules are.) 

2. Informational (e.g., I would like to know what resources are available for implementing 
the SIE modules.) 

3. Personal (e.g., I would like to know how my teaching is supposed to change using the 
SIE modules.) 

4. Management (e.g., I am concerned about my inability to manage all that the SIE modules 
require.) 

5. Consequence (e.g., I am concerned about how the SIE modules affect students.) 

6. Collaboration (e.g., I would like to develop a working relationship with both our faculty 
and outside faculty using SIE modules.) 

7. Refocusing (e.g., I would like to modify our use of the SIE modules based on the 
experiences of our students.) 

The hypothetical concerns profile in Eigure 3 illustrates how early concerns (e.g., informational) 
are the most intense early in an implementation, but decrease over time, and are replaced by 
other concerns. It is important to note that in the CBAM theory, the changes depicted in Eigure 3 
can take two to three years to be observed. 



*** HRI modified the original CBAM questionnaire to tailor it for the SIE program. 
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Hypothetical Concerns Profile at 




Type of Concern 



-♦ — Early — ■ — Middle 



Figure 3 



Figure 4 shows CBAM data from Cohort 2 teachers participating in SIE professional 
development. Based on these data, SIE was successful at addressing these teachers’ initial 
concerns about the program and the implications for classroom practice; this pattern is nearly 
identical to that observed last year among Cohort 1 teachers. 



SIE Teacher Concerns (CBAM)* 




— ♦— Pre PD — ■ — Post PD 

* Post PD profile is significantly different than the Pre PD profile, p < 0.05 (MANOVA) 

Figure 4 
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In addition, the concept of inquiry was introduced on the first day of each three-day training 
through a “gallery walk,” in which participants were asked to read a variety of definitions of 
inquiry and choose the one that most “spoke” to them. The facilitators then led a discussion, 
asking teachers to share their perspectives. Participants also experienced inquiry through an 
investigation on the first day. 

Although these activities exposed participants to ideas and examples of inquiry, neither a 
definition of inquiry and its essential components, nor SIE’s vision for how inquiry is part of 
effective science instruction was made explicit in the observed sessions. When asked to share 
their own definitions of inquiry, typical participant responses were: “hands-on activities,” “a 
higher level of learning for students,” “a shift from passive learners to active learners,” “problem 
solving,” and “students generating questions to further their learning.” Although the program 
provides teachers with the National Research Council’s “Essential Eeatures of Classroom 
Inquiry, and Their Variations,”'' it’s not clear whether participants were given sufficient 
opportunity to engage with these ideas in the time allotted. As a result, they may have completed 
the training still lacking a clear idea of inquiry-based science instruction. 

To help participants better understand the role of inquiry in their teaching of the module, the 
program may want to increase the scaffolding for this portion of the training. One possibility 
would be to first present and discuss the five essential features, without the variations, to help 
participants develop a common language and vision for inquiry. The variations, which show 
different ways these features can be incorporated into instruction, could then be introduced. 
Analyzing activities in the modules for how they incorporate these elements, a task currently in 
the training, could then serve as an opportunity for participants to apply what they are learning 
about inquiry to the materials they will use in the classroom. (It is interesting to note that most 
of the activities in the various modules fall on the “guided” side of the variations chart, but 
participants may not have recognized that fact.) Communicating a clear and consistent vision of 
inquiry would avoid confusing teachers and provide them with a concrete image on which to 
base their science teaching, as well as reducing the likelihood that teachers would see these 
activities as, in the word of one interviewed teacher, “fluff.” 

• Preparing teachers to implement the module at the mechanical level 
Eamiliarizing teachers with the structure and activities in the module is a necessary step in 
moving them along the continuum towards mechanical, and then purposeful, implementation of a 
module. Based on HRI’s observations, the trainings appeared to be very successful at 
familiarizing teachers with the materials in the modules they would be using, as well as the 
mechanics of working through the various investigations. As soon as teachers received the 
teacher manual for the module, most facilitators gave the teachers an overview of what is in the 
manual by walking them through the various sections, including the unit overview, background 
information for teachers, alignment to national standards, and assessment. 

The teachers then spent the bulk of the training experiencing the activities in the module as if 
they were students. Doing so provided them with the operational knowledge of what the 



" See page 29 in: National Research Council. (2000). Inquiry and the national science education standards. 
Washington, DC: National Academy Press. 
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materials looked like and how each investigation was supposed to work. Because the trainings 
were led by experienced teachers who had previously used these modules, participants received 
tips on managing the module materials and students throughout the training. Management tips 
included, for example, showing teachers the proper way to build a circuit with Fahnestock clips, 
suggesting that teachers have students do the streak test several times with their minerals to get 
good results, and ideas on how to empathetically discuss the death of insects used in some of the 
modules with students. 

Although most sessions were strong in this area, observation data indicated that there was some 
variation in quality. The main difference appeared to be due to the extent to which the session 
provided opportunities for teachers to reflect on how they would implement the module in their 
classrooms. Common modes of reflection included group discussions about activities from both 
the student perspective and the teacher perspective; charting facilitation tips; and creating a 
notebook of facilitation notes that the teachers could refer to during implementation. One 
observer described a session where notebooking was successfully used: 

During a three -day training on the Rocks and Minerals module, participants used their science notebooks to 
reflect on how to use the materials in the kit. During this session, the facilitators encouraged the 
participants to use one page to write what the students would have in their notebooks and one page to write 
notes that they would like to remember as teachers. For example, when the participants were completing 
the activity where they see if the minerals are reflective, one of the facilitators said, “I hear [the other 
facilitator] using this question; ‘Does this reflect like metal or reflect like glass?’ You may want to write 
that down in your teacher’s page.” 

In Other sessions, little time was allowed for participants to reflect on the use of the materials. 

For example, in some sessions, although participants were encouraged to set up their notebooks 
with student and teacher pages, they were not given adequate opportunities to step back from the 
student role and think about the activity from a teacher’s perspective. 

Overall, familiarizing teachers with the activities in the module, and providing them with 
practical implementation tips, was a strength of the Initial Module Training sessions. The SIE 
program may want to consider ways to ensure that all sessions provide both the structure and the 
time for teachers to reflect on how they can best implement the activities. 

• Deepening teachers’ understanding of the content in the module at the student level 
Although teachers can implement a module mechanically with a limited understanding of the 
relevant science content, purposeful implementation requires teachers to have a solid grasp of the 
science content in the module. As a start, teachers need to understand what concepts and skills 
each lesson is intended to teach. 

The main strategy used for deepening teachers’ understanding of the science content was having 
them complete the module activities as if they were students. This strategy is commonly used in 
professional development and makes good use of workshop time as it also serves the goal of 
familiarizing teachers with the module. To be effective, though, workshops that employ this 
strategy must also provide opportunities for the participating teachers to “step back” and reflect 
on the activity, both as a learner of scientific content and as a teacher getting ready to implement 
the activity in their classroom. 
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In nearly all of the observed sessions, facilitators began each activity with a “focus question,” 
which, as the name implies, is meant to focus participants on the main ideas of the activity. In 
the most effective sessions, facilitators had participants re-examine the focus question after 
completing the activity. For example, in training on the Human Body module, prior to engaging 
with the activity in which they constructed a model of a leg, participants were asked to consider 
how muscles enable bones to move and record their ideas in their science notebooks. After the 
activity, participants discussed what they had learned related to this question as a result of the 
exploration. Facilitators also had participants consider how their initial ideas had changed as the 
result of the investigation. 

In about half of the observed workshops, opportunities for participants to reflect on what they 
were learning were inconsistent; facilitators had participants reflect on some activities, but not 
others. For example: 

During an activity on levers from the Levers and Pulleys module, participants were focused on answering 
the question, “What happens to the effort needed to lift a load as the position of the load changes?” Groups 
tested the effort needed as the load was moved to six different locations on the lever arm. At the end of the 
activity, participants graphed their results and compared these graphs to ones they completed during an 
earlier activity in which they moved the fulcrum closer and farther from the effort. The facilitators then 
engaged the participants in a discussion to help them make sense of the content. However, this focused 
sense making was observed only for this one activity. Although some of the other activities included 
reflection questions, they tended to be too vague to ensure that teachers really understood the content (e.g., 
“What did you learn about pulleys?”). 

In some cases, participants were able to draw a conclusion from an activity, but were not helped 
to see the connection between their specific conclusions and the scientific principle targeted by 
the activity. For example, in an activity from Environments, participants did an activity in which 
they observed whether beetles migrate toward the moist or dry side of a container. However, the 
connection was never made to the broader scientific idea of “preferred environment.” Finally, in 
a small number of sessions, observers noted that teachers had no opportunities at all to reflect on 
the content within the modules. In these sessions, the participants were focused solely on the 
mechanics of the activities. 

In addition to variability in the extent of opportunities provided for teachers to reflect on the 
content, sessions varied in the extent of accuracy of the content being presented to teachers. 
Many sessions handled the science content well, but others had content issues that ranged from 
minor to substantial. For example, in a session on the Ecosystems module, during a discussion 
about how oxygen is released during photosynthesis and carbon dioxide during respiration, one 
facilitator said “plants make much more oxygen during the day than what they use at night.” In 
the context of the discussion, the statement may have given participants the impression that 
respiration occurs only at night. Although the facilitator clearly knew that respiration occurs 
constantly, day and night, the “loose language” could well have reinforced a common 
misconception among the less knowledgeable participants. 

Sometimes content issues arose because the examples used to illustrate a concept were more 
complicated than the participants or the facilitators realized. For example, in a Chemical Tests 
workshop, participants were engaged in a discussion about the differences between solutions and 
suspensions. Examples of a number of common mixtures, such as a mixture of baking soda and 
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water, toothpaste, and Jell-0, were introduced to try to help participants understand the 
distinctions. Unfortunately, these examples are not as straightforward as they appear to be and 
led to a number of inaccurate statements being made. For example, in discussing the examples 
the facilitator said, “Toothpaste is a solution unless it’s got globs of that green freshener mixed in 
it.” and “Jell-0 is a solution; add fruit it becomes a suspension.” In actuality, toothpaste and Jell- 
O are both colloids (a concept that is not covered in the module and consequently was not dealt 
with in the training). Because these mixtures are ones that students are likely to bring up, this 
example highlights the importance of teachers developing an understanding of the science 
content at a deeper level than students are expected to learn. The example highlights the need for 
facilitators to have deep understanding of the content as well. 

In some sessions, facilitators dealt with science content beyond the student level, though they did 
not always make clear the distinction between content meant for students and content meant for 
the teaching participants. For example, in a Magnets and Motors session, facilitators described 
how the gauge of a wire is related to its electrical resistance. Later in the session, the facilitators 
noted that there are three types of metals that are magnetic: iron, nickel, and cobalt. However, in 
neither case was it made clear to the participants whether or not they should introduce this 
information in their classroom. Pointing out to participants what content information is for their 
own benefit and which is appropriate for students would avoid confusion about what content was 
developmentally appropriate for students. 

Although it is not feasible, or desirable given the advantages of spaced learning, to engage 
participants with all of the science content around a topic in an introductory workshop, it is 
important that a solid foundation be laid for future trainings to build upon. The program may 
find it beneficial to create a conceptual map for each topic that includes not only the ideas 
covered in a module, but also related scientific ideas that are necessary for a more complete 
understanding of the content and that help tie the module ideas together. Doing so may help 
facilitators decide when participants need to be introduced to ideas beyond the student level. 

In addition, the SIE program may want to consider incorporating a mechanism into the initial 
module trainings to help participants begin to see the interconnections of the concepts. One 
possibility would be to have the participants, after completing each module activity, come to a 
consensus statement about what students are supposed to learn from the activity. At the end of 
the workshop, the facilitators could show the participants how the ideas fit together from a 
scientific standpoint. Such activities would help participants begin to see the big picture, and 
help prepare them to understand the content storyline in the module. In addition, providing this 
conceptual map to participants would give them something to refer back to while they are 
teaching the module. 

• Deepen teachers' understanding of the content storyline of the module 
Purposeful implementation of a module requires teachers to have a good grasp of how the 
targeted science ideas are developed through the activities in the module. Specifically, teachers 
need to understand both what concepts and skills each lesson is intended to teach and how each 
lesson contributes to the development of this knowledge. As described previously, a solid 
understanding of the scientific content (concepts and skills) addressed in the module is a 
prerequisite for understanding the storyline. 
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The modules themselves provide some background information for teachers on the content they 
intend students to learn through the module activities. For example, the STC modules provide an 
overall introduction to the module, a rough “storyline,” (though the STC storylines tend to focus 
more on what students will do than on what students will learn) and a brief content overview for 
teachers before each lesson. Similarly, the FOSS modules provide a unit overview and a teacher 
background section for each investigation. However, the modules vary in the extent to which 
they explicitly relate the student activities to this content information. Helping teachers 
understand the relationships between activities and the concepts targeted in a module is essential 
in helping them move toward purposeful implementation of a module. 

The content storyline element of the professional development was rarely seen in the observed 
sessions. In most of the observed sessions, facilitators pointed out the section of the module that 
described the content covered, but few sessions went beyond this step. 

If the SIE program decides to introduce a conceptual map to participants at the training, one 
possibility for getting participants to think about the content storyline would be to engage them 
with the connections between the conceptual map and the module activities. For example, 
participants could be split into small groups, with each group asked to trace the development of a 
different idea through the module. Small groups could then share their findings with the whole 
group. (It is important to note that this type of activity would likely need to take on a different 
flavor for the early elementary grade modules, as those focus less on content and more on 
developing students’ process skills.) 

However, before engaging participants with this type of task, it will be important for SIE 
facilitators to develop the conceptual map and storylines themselves. The SIE program may 
want to have these conceptual maps reviewed by content experts (e.g., Ph.D. scientists in the 
appropriate field) to ensure they are both accurate and cohesive. In the future, the program could 
build on these storyline documents, adding information about common student misconceptions 
and how they can be addressed with the module. In addition to the benefit these documents 
would have for participating teachers, they would also be of great help in bringing new 
facilitators up to speed. 

• Preparing teachers to use instructional strategies, specifically notebooking and 
questioning, that support module implementation 
Teachers need to understand the pedagogical strategies that can be used in implementing a 
module effectively, as well as when and why to use each of those strategies. Overall, the SIE 
trainings appear to have provided teachers with many opportunities to learn the instructional 
strategies they will need to implement the module. Many of the facilitators had experience 
implementing the modules in their own classrooms, which allowed them to share teaching 
strategies that they know work. (This aspect of the sessions was greatly appreciated by 
participants.) In addition, facilitators were often very explicit about what they were doing and 
why. Eor example, facilitators stopped frequently and pointed out the different ways they were 
utilizing science notebooks, an important way to integrate literacy skills into science teaching, so 
participants could use those notebooking strategies with their students. 
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Facilitators also did a nice job of sharing questioning strategies with participants. One effective 
approach used was when the facilitators had participants step out of “learner mode” and into 
“teacher mode.” In these cases, facilitators explicitly pointed out the questions they used and 
why they were using them. Interestingly, observers rarely noted facilitators highlighting the 
questions that the modules suggest teachers use. Making explicit why or why not they should 
use the questions suggested by the module may help participants be more effective with the 
modules. 



Module Enrichment Trainings 

In Year Two, Cohort I teachers were offered a two-day module enrichment workshop on their 
initial module. The module enrichment workshops are intended to move teachers further along 
the continuum from mechanical to purposeful use of the SIE-supported modules by deepening 
teachers’ understanding of the science content and content storyline of the module, and by 
deepening their pedagogical content knowledge. Specifically, the project’s goals for the 
enrichment workshops were to: 

1. Debrief experiences from teaching the module; 

2. Uncover and understand the big ideas in the module; 

3. Recognize and interpret student thinking and misconceptions (about the targeted content); 

4. Understand how the tasks within the module connect and build to the big ideas; and 

5. Identify real-world connections to the big ideas. 

In the Fall of 2007, the SIE program piloted four module enrichment workshops. HRI observed 
all four of the pilot module enrichment workshops in the fall of 2007 and 6 of the 38 module 
enrichment workshops offered in the spring of 2008. About half of Cohort 1 teachers eligible to 
attend a module enrichment workshop did so. Because many schools were concerned about the 
amount of time teachers were expected to be out of the classroom, the program decided to make 
the module enrichment optional this year and will provide opportunities for teachers to attend a 
module enrichment workshop next year after their second implementation of the module. This 
section of the report describes the quality of the SIE module enrichment professional 
development using data from teacher questionnaires, teacher interviews, and HRI’s observations. 

Teachers’ Perceptions of the Quality of the Module Enrichment Trainings 
Teachers had somewhat mixed opinions about the SIE module enrichment workshops. At the 
end of each session a questionnaire was administered that included questions about the quality of 
the workshop. As can be seen in Table 11, nearly all participants thought that facilitators 
encouraged active participation (94 percent) and that there were opportunities for teachers to 
collaborate with peers (93 percent). 
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Table 11 



Teacher Opinions of the Quality of SIE Module Enrichment Professional Development 





Percent Agreeing^ 
(N = 590) 


The facilitators encouraged active participation and investigation by all participants 


94 


There were opportunities for participants to express their views and collaborate with peers 


93 


The facilitators were well prepared 


90 


Questions about the science content in the module were adequately addressed by the facilitators 


87 


Adequate, time, structure, and guidance were provided for participants to discuss the SIE modules 




and pedagogical strategies with each other 


87 


Adequate time, structure, and guidance were provided for participants to reflect individually on the 




substance of the professional development 


86 


The facilitators shared tips and suggestions for successfully implementing the module 


85 


The professional development reflected careful planning and organization 


85 


The facilitators modeled effective teaching strategies 


85 


The goals of the professional development were clear 


81 


The professional development increased my understanding of the science content in the module 


78 


The professional development prepared me to use the teaching strategies promoted by SIE 


78 


The facilitators explicitly discussed how, when and why to use different teaching strategies 


77 


The various components of the training were useful in meeting the goals of this professional 




development 


76 


The professional development was relevant to my classroom instruction 


76 


The professional development increased my confidence in my ability to teach using the SIE module 


75 


The professional development familiarized me with the activities and materials in the module 


73 


The professional development was worth the time that I invested 


65 


I would recommend this professional development to a colleague 


64 



Includes those teachers agreeing or strongly agreeing with each statement. 



However, only about two-thirds of respondents thought the workshop was worth the time 
invested (65 percent) or that they would recommend the workshop to colleagues (64 percent). 
Nearly one-third of the attendees thought the pace of the workshop was too slow. (See Table 
12 .) 



Table 12 



Teacher Opinions of the Pace of SIE Module Enrichment Trainings 





Percent of Teachers 
(N = 590) 


Much too slow 


9 


Somewhat too slow 


23 


Appropriate 


67 


Somewhat too fast 


1 


Much too fast 


0 



Participants were also asked to reflect on which aspects of the workshop they found to be the 
most valuable. The most frequent responses identified opportunities to improve their content 
understanding (27 percent) and to collaborate with peers (26 percent) as being particularly 
beneficial. Interview responses about the most valuable aspects of the workshops frequently 
focused on the content presented. For example: 
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Finding out more about [ the topic], I feel more comfortable teaching this to my students. 

The content deepening; prior, students would have questions about things [ that I] 
couldn 't answer. 

[The module enrichment workshop] did help us, quite a bit with the content because, like 
I said, we were a little bit scattered as to how this was all going to come together. ..but 
after we went to the content deepening, we understood a lot more. 

Other responses included comments about the high quality of facilitation and teaching strategies 
gained. 

In another open-ended item on the questionnaire, participants were asked to reflect on aspects of 
the workshop that they thought could be improved upon. Thirty-four percent of the respondents 
indicated that they did not think the workshop needed any improvement. Of the suggestions for 
improvement that were offered, the most common suggestion, given by 19 percent of the 
respondents, was to condense the workshop into a shorter program. No other suggestion was 
mentioned by more than a few participants. 

Although the questionnaire data were quite positive, interviewed teachers expressed more 
concern about the quality of these workshops. Interviewees indicated a general sense of 
confusion about the purpose of the module enrichment workshops, indicating they didn’t know 
ahead of time what to expect from the workshop or that the expectations they had were not met. 
As one teacher said: 

We went there thinking it was going to be different than what it was. We went thinking 
that. . . they were going to give us more activities, like, for the kids, or more. . . books, or. . . 
other reference sources that we could go to. 

Two interviewed teachers indicated that they decided not to attend a module enrichment 
workshop because of negative feedback from teachers who had attended; a teacher scheduled to 
attend an enrichment workshop expressed a concern based on colleagues’ experiences: 

We have had some colleagues that have gone to the enrichment deepening, or the content 
deepening, and they have come back not real happy... because they said that [it] really 
was not relevant to classroom use. So, actually, Fm a little... skeptical now, 
about. ..going to my content deepening. And I think the content deepening that my 
colleagues have gone to has been more of a deepening for them, more of an 
understanding of the module, but it wasn’t something that they felt they could actually 
bring back and use in the classroom. 

Clarifying the purposes of the module enrichment workshops for teachers and making sure those 
purposes are evident in the workshop will be important as the program prepares to offer more of 
these trainings in the future. 
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Observers’ Perceptions of the Quality of the Module Enrichment Trainings 

The analysis of observation data from the module enrichment trainings focuses on the major 

goals of the sessions. 

• Debriefing experiences from implementing the module in the classroom. 

Most teachers who attended module enrichment workshops had already attended an initial 
module training and taught the module at least once. Typically, teachers who are implementing 
a new program have a strong desire to share their initial experiences, so much so that they are 
often not able to focus on anything else until this desire has been satisfied. Thus, SIE wisely 
incorporated a session in the spring module enrichment workshops to provide this opportunity to 
participants. During this session, facilitators posed five questions to participants: 

1 . What strategies did you use to implement inquiry in the module? 

2. How were you able to integrate the use of a notebook in your module instruction? 

3. What are some assessment strategies (formative/summative) you have used with the 
module? 

4. What questions concerns and comments do you have regarding management of the 
modules (i.e., time, classroom, and materials)? 

5. What science concepts and content misconceptions did the students present to you? 

Participants provided their thoughts, either individually or in small groups, and then engaged in a 
whole group discussion about each question. Although this portion of the workshop tended to 
work well, observers noted two areas in which it could be improved. Most of the teachers did 
not appear to be ready for some of the questions, particularly the ones on assessment strategies 
and misconceptions. In order to identify what misconceptions students have, teachers need to 
have both a strong understanding of the content so they know which ideas are correct/incorrect, 
and an understanding of which incorrect ideas get in the way of student learning. Lacking this 
knowledge, participants tended to misinterpret the questions. For example, in one session, 
participants interpreted the question about student misconceptions as “what did students have 
trouble learning.” 

The second issue identified in the observations was that too much time was devoted to this 
segment of the workshop. Several observers noted that participants tended to get off task and 
start talking about unrelated topics in their small group discussions. In addition, during the large 
group discussion participants often repeated the points already mentioned by others. 

The program may want to consider reducing the number of questions posed to participants, 
focusing on those that are most salient given their previous experiences, and tightening the 
schedule to keep the pace appropriate. 

• Understanding the big ideas of the module 

One of the goals of the module enrichment workshops is to help teachers understand the science 
content in their module at a deeper level. In observed workshops, the science content was 
addressed in different ways. In some workshops teachers were engaged with hands-on activities; 
in other sessions participants read articles about the content. Some facilitators presented the 
content to participants in a lecture format. It was not uncommon for workshops to use a 



Horizon Research, Inc. 



35 



October 2008 




combination of these approaches. Across the observed sessions, there was variability in the 
quality of content treatment regardless of the instructional strategies utilized. 

In most cases, teachers were engaged with important scientific ideas, evident by their 
conversations with each other and with facilitators. However, across the observed sessions many 
teachers still seemed to lack the big picture of the topic; the disciplinary content often came 
across as isolated bits of information, with not enough emphasis on developing the larger 
conceptual framework. In cases when a session did address the larger framework, participants 
sometimes struggled because they did not yet have all the pieces necessary to see the underlying 
connections among the concepts and how they related to the big ideas. For example, in one 
workshop the facilitators elected to skip a concept that they thought was not that important to 
cover. However, later in that workshop, participants had difficulty understanding some of the 
ideas that were covered because they did not understand the more basic idea that was omitted. 

In addition, participants in a number of the observed workshops indicated that they did not 
understand why they were being asked to learn ideas that they were not expected to teach to their 
students — they did not see how the broader ideas could help them be more effective in teaching 
the module to their students. Given that most elementary teachers do not have strong science 
backgrounds, helping teachers develop a working understanding of the content in their module 
and the broader knowledge they need to be effective in the classroom may be the most difficult 
aspect of the program’s work. As noted earlier, developing conceptual maps for the science 
topics related to each module may help teachers see the relationship among the ideas that are 
addressed directly in the module and those which will help them implement the module more 
effectively. 

• Understanding how the tasks within the module connect and build to the big ideas 
( Content Storyline) 

In all observed sessions, the content storyline was addressed with a similar activity. Participants 
were given a list of statements and asked to categorize them as a “big idea,” “learning goal,” or 
“instructional task.” Once participants had categorized the statements, they were asked to 
connect the statements in a hierarchical way to create a “map.” Though the activity was similar 
across all observed sessions, the nature and quality of implementation varied. For example, in 
some sessions participants were provided with definitions of big idea, learning goal and 
instructional task; in other sessions participants were asked to come up with the meanings on 
their own. In one session when participants asked the facilitator for more guidance about what 
each category meant, the facilitator encouraged them to come up with their own definitions. In 
sessions where the terms were not defined, observers noted that several participants became 
frustrated and disengaged from the activity. In other cases, the lack of agreement on the meaning 
of these terms got in the way of the participants focusing on the intended goal of the activity. In 
the absence of understanding the meaning of the categories, participants’ focus shifted from 
attending to how key ideas are developed and addressed in the module to developing a definition 
for each category. 

At the end of the activity, facilitators typically presented an answer “key,” and participants’ maps 
were usually different than the key. Facilitators told participants that it did not matter how they 
arranged their statements, as long as they had a good argument to support their reasoning. 
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Avoiding a definitive “right” answer appeared to generate a great deal of confusion and 
disagreement among participants about which statements fit into which category. For example, 
in one session, participants and facilitators had an extensive conversation about how each 
statement was categorized. Participants ultimately disagreed with the facilitator and could not be 
won over. This discussion dominated the session and many other participants were having side- 
bar conversations that were unrelated to the topic. Once the facilitator got everyone back on 
track, the group that did not agree with how the statements were categorized disengaged from the 
activity, stating that they just “don’t get it.” One observer noted, “Only a small number of 
participants contributed; the participants that contributed seemed to have the best understanding 
of the content.” 

Although this activity may have helped participants understand how the ideas in a module 
connect, it was not apparent that participants always understood how the module developed those 
ideas. Observers noted that although there were instructional tasks included in the activity, there 
typically were no explicit links made between the ideas and specific lessons in a module. 

• Recognize and interpret student thinking and misconceptions 
(Pedagogical Content Knowledge) 

In addition to understanding the science content related to a module, teachers would benefit from 
knowing what ideas students are likely to have that interfere with their learning, how to assess 
which ideas (correct and incorrect) their students have, and what strategies they can use to move 
student thinking forward. This type of knowledge is often referred to as pedagogical content 
knowledge (PCK), and generally requires an understanding of both the science content and how 
students learn a particular idea. 

The observed module enrichment workshops took different approaches to deepening teachers’ 
PCK. In some sessions participants examined samples of student work to identify student 
thinking and misconceptions about the targeted content. In other sessions participants watched 
videos of student interviews that highlighted student thinking and their misconceptions about the 
targeted content. Participants were also given publications about formative assessment to read in 
several sessions. 

Overall, this portion of the workshop seemed aimed more at introducing teachers to the idea that 
students have misconceptions and that there are strategies for assessing student understanding 
than at preparing teachers to implement these strategies in their classroom. Given that many of 
the teachers had not previously encountered these ideas, it would not be reasonable to expect 
much more. Not surprisingly, given the weak content backgrounds of many elementary teachers, 
some participants struggled with this portion of the workshop. For example, in one session 
participants had difficulty identifying the misconceptions in samples of student work. The 
observer noted, “Participants did not understand the science content well enough to fully 
understand what student thinking was scientifically accurate and what thinking was inaccurate.” 

In other cases, participants had issues with the activity because it was too disconnected from the 
module. For example, in some sessions, the samples of student work they were given to analyze 
were not from an activity in the module. In another session, participants were able to identify the 
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misconceptions in the videotaped student interviews, but were unable to come up with strategies 
on their own to help students let go of those misconceptions. 

Although this portion of the workshop successfully introduced teachers to the concept of 
formative assessment and raised their awareness of misconceptions, preparing them to use this 
information in their classroom would likely take much more intensive professional development. 
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Impact of SIE on Teachers and Teaching 



This section of the report focuses on how the SIE program has impacted teachers, including their 
preparedness to use their SIE-provided module, and their classroom teaching. Data come from 
teacher questionnaires and interviews, assessments of teacher content knowledge, and classroom 
observations during module implementation. 



Teacher Preparedness 

The post-professional development questionnaire included a series of items aimed at assessing 
teachers’ perceptions of their preparedness to use the teaching strategies promoted by SIE (e.g., 
how well prepared they felt to use the inquiry-based teaching strategies embedded in the SIE 
module, use science notebooks to support student learning of the content in the SIE module, and 
use a learning cycle). To assess impacts, teachers were asked about both their current 
preparedness and their preparedness prior to the professional development session. This 
“retrospective pre” approach is useful when respondents are likely to change their perceptions of 
initial knowledge/preparedness as they learn more about the topic (i.e., they didn’t realize how 
much/little they knew about a topic until after their participation in the program). Cohort 2 
teachers were also asked these questions on the end-of-year questionnaire to examine whether 
use of the module affected teachers’ perceptions of preparedness. 

Responses to these items were combined into a composite variable (called “perceptions of 

12 

pedagogical preparedness”) to reduce the unreliability associated with individual survey items. 
Each composite has a minimum possible score of 0 and a maximum possible score of 100. A 
score of 0 would indicate that a teacher selected the lowest response option for each item in the 
composite, whereas a score of 100 would indicate that a teacher selected the highest response 
option for each item. 

These longitudinal data have a nested structure, with time points nested within individual 
teachers. Statistical techniques that do not account for such nested data structures can lead to 
incorrect estimates of the relationship between independent factors and the outcome. 

Hierarchical regression modeling*^ is an appropriate technique for analyzing nested data and was 
used to examine trends in teachers’ composite scores. 

As can be seen in Eigure 5, teachers’ perceptions of their pedagogical preparedness increased 
dramatically as a result of the professional development. Composite scores for Cohort 1 teachers 
increased from a mean of 5 1 to a mean of 77 as a result of the initial training on their second 



Definitions of this and other composites described in this report, a description of how the composites were 
created, and reliability information are included in Appendix C. 

Bryk, A.S. & Raudenbush, S.W. (1992). Hierarchical linear models: Applications and data analysis methods. 
Newbury Park, CA: Sage Publications. 
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module (a large effect^"^ of 1.16 standard deviations). Similarly, scores of teachers who attended 
the module enrichment trainings changed from a mean of 57 to a mean of 74 (an effect of 0.77 
standard deviations). Cohort 2 teachers’ scores were similar to Cohort 1 teachers’ scores in Year 
One, increasing from 26 points to 73 points as a result of the initial module training (a very large 
effect of 2.18 standard deviations). 

It is interesting to note that Cohort 1 teachers’ two retrospective pre scores are similar, and that 
their perceptions of preparedness prior to the module enrichment workshop did not reflect the 
gains from the initial training on their second module (which most of the teachers had attended 
several months before the module enrichment sessions). This result may be due to the teachers 
answering the questions with respect to two different modules. The apparent difference in 
retrospective pre scores between the two cohorts is likely due to Cohort 1 teachers having 
participated in training last year. 
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Figure 5 



The slight, but significant, decrease at the end of the school year for Cohort 2 teachers may be 
due to a wearing off of a “halo effect” (i.e., teachers who enjoy a professional development 
session may overestimate the impacts of that session when asked immediately following it). The 
dip may also be due to teachers having the experience of trying these strategies in the classroom, 
and realizing they weren’t quite as well prepared as they thought they were. Still, the overall 
trend is positive with large increases for both cohorts, an indication that the professional 



The effect size for the comparison of two adjusted means is calculated as the difference between the means 
divided by the pooled standard deviation. Effect sizes of about 0.2 are typically considered small, 0.5 medium, and 
0.8 large. Cohen, J. (1988). Statistical power analysis for the behavioral sciences. Hillsdale, NJ: Lawrence 
Erlbaum Associates. 
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development increased teachers’ feelings of preparedness to implement the modules in their 
classrooms. 

The questionnaires also asked teachers to rate their understanding of the science content in their 
module. Items asked about their understanding of (a) the student learning goals in the SIE 
module; (b) the science content in the SIE module at a deeper level than what students are 
expected to learn; and (c) ideas (either correct or incorrect) that students are likely to have about 
the content prior to instruction. These items were also combined into a composite, in this case 
titled “perceptions of pedagogical content knowledge.” 

The results for this composite parallel those for the perceptions of pedagogical preparedness 
composite, likely for the same reasons. (See Eigure 6.) The increases for Cohort 1 teachers 
correspond to large effects of 1.46 and 0.93 standard deviations for the initial and module 
enrichment trainings, respectively. The 28-point increase for Cohort 2 teachers corresponds to a 
very large effect of 2.18 standard deviations. As with the perceptions of pedagogical 
preparedness composite, there was a small, significant decrease from just after the professional 
development to the end of the school year for Cohort 2 teachers. 



Teacher Perceptions of Their Pedagogicai Content 
Knowiedge* 



Vi 

c 100 1 

o I 




Cohort 1 Initial PD, Cohort 1 Module Cohort 2 Initial PD, 

2nd Module Enrichment, 1st 1st module 

module 



□ Retrospective Pre □ Post-PD ■ End of Year 



* Teacher scores changed significantly over time, p < 0.05 (HLM) 

Figure 6 



In interviews, teachers indicated that the professional development helped them understand the 
content in their module. Eor example: 

It did help us, quite a bit with the content because, we were a little bit scattered as to how 
this was all going to come together, but after we went to the content deepening, we 
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understood a lot more. And, we understood the actual skills and concepts that they need 
to know when they go on to grade. 

In spring 2008, HRI conducted a pilot study of the impact of SIE’s “Module Enrichment” 
workshops on teacher science content knowledge. This pilot study focused on four SIE- 
supported modules: Electric Circuits, Levers and Pulleys, Mixtures and Solutions, and Motion 
and Design. HRI developed a short (15 items) teacher assessment for each of these modules. 
The assessment items were reviewed both by the Pennsylvania Department of Education and by 
external science content experts. 

Cronbach’s Coefficient Alpha was calculated to examine the extent to which the items on each 
assessment appeared to be measuring the same construct; items that did not correlate well with 
the overall scale were dropped from the analysis. Table 13 shows the reliability for each group 
of items for the modules that were assessed. Three of the 4 assessments had acceptable 
reliabilities. The reliability for the Motion and Design assessment was fairly low; thus, results 
from this assessment should be interpreted with caution. 



Table 13 



Teacher Assessment Item Reliability 





Number of Items 


Reliability 


Electric Circuits 


12 


0.61 


Levers and Pulleys 


14 


0.78 


Mixtures and Solutions 


15 


0.71 


Motion and Design 


13 


0.56 



The assessments were administered at the beginning and the end of the two-day module 
enrichment workshops on these modules. HRI received pre- and post-test responses from 124 
teachers across the four modules; however, 22 teachers did not indicate the topic of the 
workshop, reducing the number of valid assessments to 102. Table 14 shows the number of 
valid responses for each module. It should be noted that the sample size for Mixtures and 
Solutions is very small, making it very unlikely a statistical test would detect a change from pre- 
to post-test, even if one truly existed. 



Table 14 



Sample Sizes for the Teacher Assessments 



Module 


Number of Valid 
Pre/Post Responses 


Electric Circuits 


38 


Levers and Pulleys 


25 


Mixtures and Solutions 


11 


Motion and Design 


28 



A Cronbach’s Alpha of 0.6 is considered acceptable, 0.7 fair, 0.8 good, and 0.9 excellent. 
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Figure 7 shows the average pre- and post-test score for each module. Post-test scores were 
significantly higher in 3 of the 4 workshops: Electric Circuits, Levers and Pulleys, and Motion 
and Design. Although the trend is in the expected direction for Mixtures and Solutions, the 
difference was not statistically significant (again, likely due to the very small sample size). 



Impact of Module Enrichment Sessions on Teacher Content 
Knowledge, by Module 




□ Pre □ Post 

* Post-test score significantly higher than pre-test score (one-tailed dependent samples t-test, p < 0.05). 

Figure 7 



The 10-point increase in scores for Electric Circuits corresponds to a medium effect size of 0.60 
standard deviations; the 20-point increase for Levers and Pulleys corresponds to a large effect of 
0.96 standard deviations; and the 9-point increase for Motion and Design is equivalent to a 
medium effect size of 0.59 standard deviations. 

HRI examined the assessment results in more detail to determine which ideas teachers seemed to 
be getting and which they still struggled with after the module enrichment workshops. On the 
Electric Circuits assessment, teachers appeared to have consistent gains on understanding what 
is and isn’t a complete circuit, and the relative brightness of bulbs in simple (i.e., single bulb and 
battery) and parallel circuits. Teachers also seemed to gain an understanding of how electric 
current flows in simple and series circuits, but did not seem to understand current flow in parallel 
circuits. 

Teachers in the Levers and Pulleys module enrichment sessions appeared to gain a better 
understanding of the definition of “effort,” identifying the different classes of levers, and 
comparing the mechanical advantage between lever and pulley systems. Teachers seemed to 
struggle with the concept of work and how it applies to levers and pulleys. 



Horizon Research, Inc. 



43 



October 2008 



Teachers in the Mixtures and Solutions sessions appear to have increased their understanding of 
the similarities and differences of mixtures or solutions. However, many teachers did not appear 
to understand what a chemical reaction is or that chemical reactions always result in new 
substances being formed. 

Teachers in the Motion and Design module enrichment sessions showed gains on the items 
related to kinetic and potential energy, and how energy is transferred. Teachers also seemed to 
gain a better understanding of Newton’s First Law (i.e., that the motion of an object will not 
change unless an unbalanced force acts on it). However, a substantial number of teachers still 
had difficulty letting go of the incorrect ideas that a forward force is needed to keep an object in 
constant motion, and that a moving object has a force in it that keeps it moving. Participants also 
seemed to struggle with the concept of friction and the direction in which it acts. A fair number 
of teachers also continued to have difficulty with the distinction between acceleration and speed, 
and consequently Newton’s Second Law (i.e., the acceleration of an object depends on both its 
mass and the net force applied to it). 



Module Implementation 

In its first year, SIE offered support for 1 1 different modules to Cohort 1 schools. In grades K-2, 
SIE offered one module at each grade. In each grade 3-6, SIE allowed schools to select one of 
two modules. In Year Two, SIE introduced a second set of 1 1 modules, including one at each 
grade K-2, and two at each grade 3-6. Cohort 1 schools were expected to implement a second 
module at each grade, and were allowed to select their new modules from among the 22 modules 
being supported by SIE. Cohort 2 schools were expected to implement one module at each 
grade, and were allowed to select from the initial set offered by SIE. 

Both teachers and principals were asked on questionnaires about factors that may have affected, 
positively or negatively, teachers’ use of the module. Results from these questions are shown in 
Table 15 and Table 16. Nearly all of the responding teachers and principals thought that the SIE 
professional development facilitated use of the module, and that teachers supported each other in 
the use of the module. Most respondents also indicated that the SOS facilitated use of the 
module. In addition, although the SIE program does not have any formal mechanisms for 
assisting teachers outside of the workshops, roughly one-quarter of the teachers and more than 
half of principals indicated that teachers received extra assistance that helped them implement 
their module. 
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Table 15 

Teacher Opinions of Factors Affecting Their Use of the SIE-Provided Module 





Percent of Teachers 
Agreeing^ 

(N = 1,768) 


Factors Facilitating Use 




The training I received from SIE made it easier for me to use the modules. 


95 


Other teachers in my school provided a support system for use of the modules. 


86 


My own science background was helpful when I was teaching from the modules. 


77 


My SOS facilitated my use of the modules. 


73 


I received assistance from SIE outside of the workshop that helped me to use the modules. 


26 


Factors Inhibiting Use 




The amount of time required to prepare for instruction with the SIE modules was problematic. 


55 


There was not enough instructional time for science to effectively use the SIE modules. 


52 


The pressures to teach mathematics and/or reading inhibited my use of the modules. 


51 


I am not able to cover all of the science topics I am suppose to because of the time it took to 




implement an SIE modules. 


43 


I did not have the modules for long enough to use as much of them as I wanted to. 


29 


The amount of time I was required to be out of the classroom was problematic. 


26 


My lack of experience in science made it more difficult for me to teach from the modules. 


15 



Includes those teachers agreeing or strongly agreeing with each statement. 



Table 16 



Principal Opinions of Factors Affecting 
Teacher Implementation of the SIE-Provided Modules 





Percent of Principals Agreeing^ 




Overall 
(N = 93) 


Cohort 1 
(N = 48) 


Cohort 2 
(N = 45) 


Factors Facilitating Use 

The training teachers received from SIE made it easier for them to use the 
modules 


99 


98 


100 


Teachers provided each other with a support system for using the modules 


98 


96 


100 


Our Support on Site (SOS) facilitated teachers’ use of the modules 


85 


83 


87 


Teachers received assistance from SIE outside of the workshop that helped 
them to use the modules 


61 


67 


56 


Factors Inhibiting Use 

The amount of time teachers were required to be out of the classroom was 
problematic 


57 


56 


57 


The pressures to teach mathematics and/or reading inhibited teachers’ use of 
the modules 


38 


38 


38 


Teachers were not able to cover all of the science topics they were supposed 
to because of the time it took to implement an SIE module 


30 


33 


27 


There was not enough science instructional time for teachers to complete the 
SIE modules 


26 


27 


24 


The amount of time required to prepare for instruction with the SIE modules 
was problematic for teachers 


23 


23 


22 


Teachers did not have the modules for enough time to complete all of the 
lessons 


19 


19 


20 


Teachers’ lack of experience in science made it more difficult for them to 
teach from the modules 


12 


10 


16 


Teachers’ lack of science content knowledge made it more difficult for them 
to teach from the modules 


12 


8 


13 



Includes those principals agreeing or strongly agreeing with each statement. 
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In terms of factors that inhibited use of the module, time was a concern of both teachers and 
principals. Principals were mainly concerned with the time that teachers were out of the 
classroom to attend professional development. In contrast, highlighted as problematic the 
amount of time required to prepare for instruction with a module; the lack of instructional time to 
use the modules effectively; and the pressure to teach mathematics and/or reading. These 
sentiments were echoed in teachers’ responses to an open-ended item that asked participants 
what aspects of the SIE program could have been improved. Comments included: 

I did not have enough prep time to prepare for instruction using the modules. 

Time was an issue for us... The other teachers [also] said how much time it took. There 
were actually some days, we would not do [the module]; we would kind of double up, 
maybe an hour another day. You know, once you get that stuff passed out, it takes a lot of 
time to, read, collect everything and, time was an issue. ..We had to work off best we 
could. 

As part of the student achievement study (described more fully later in this report),*^ teachers 
complete a “module use” survey that asked them to indicate which lessons from the module they 
used in their teaching, how much instructional time was devoted to the module, and the extent to 
which they supplemented the module with other instructional materials. As part of the student 
achievement study, HRI examined how module use changed from Cohort 1 teachers’ first 
implementation in Spring 2007 to their second implementation in Fall 2007. Although the 
sample size is too small to statistically examine whether extent of module use differs between 
first and second implementation, a descriptive analysis of the data (see Table 17) reveals that on 
average, teachers using 4 of the 6 modules {Levers and Pulleys, Motion and Design, Mixtures 
and Solutions, and Variables) devoted much more instructional time to the module during their 
second implementation than during their first implementation. Interestingly, the change in 
instructional time varies considerably by module, with those aimed primarily at 5* and 6* grade 
having the largest increases. The percent of the module covered does not appear to have 
changed from first to second implementation in most cases. 



The student achievement study focused on 6 of the 1 1 modules provided by SIE in Year Two as described later in 
this report. Data from the Year Two Fall assessment administration were available for this report; data from the 
Spring assessment administration will be reported in an addendum report. 
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Table 17 



Descriptive Statistics for Instructional Variables, by Teacher Experience with the Module 





Percent of Lessons 
Covered 


Instructional Time on 
Topic in minutes 


Percent of Instructional 
Time Based on Module 


Mean 


Standard 

Deviation 


Mean 


Standard 

Deviation 


Mean 


Standard 

Deviation 


Electric Circuits 














First Time 


86.37 


16.69 


771.43 


285.23 


92.86 


18.05 


Second Time 


89.09 


12.60 


786.40 


307.11 


97.56 


4.90 


Levers and Pulleys 














First Time 


71.49 


26.19 


590.22 


451.29 


93.69 


11.23 


Second Time 


85.03 


13.87 


1,107.91 


539.33 


94.64 


10.09 


Mixtures and Solutions 














First Time 


75.13 


21.38 


842.44 


365.12 


93.62 


18.29 


Second Time 


86.10 


13.76 


1,119.94 


530.60 


85.94 


21.14 


Motion and Design 














First Time 


83.21 


23.49 


641.90 


277.38 


86.20 


25.13 


Second Time 


83.11 


22.92 


961.42 


528.06 


94.44 


13.70 


Rocks and Minerals 














First Time 


89.14 


20.13 


818.42 


261.02 


94.10 


10.11 


Second Time 


86.49 


23.56 


710.56 


285.40 


91.09 


20.31 


Variables 














First Time 


86.33 


15.82 


807.69 


304.43 


99.29 


2.75 


Second Time 


84.36 


25.96 


1,228.57 


753.25 


95.00 


7.56 



The increase in the amount of time teachers spent on the modules may be due, in part, simply to 
the fact they received the modules in Year Two in the Fall. The PSSAs are administered in the 
Spring, reducing the amount of instructional time teachers have to implement the module (both 
because of the days for administering the assessments and the days schools spend preparing 
students for the tests). In addition, as a result of their experiences in Year One, some schools 
have built more time for science into their schedules. As one teacher stated in an interview when 
asked what led to changes in implementation from Year One to Year Two: 

~We had more time. ..since last year was our new year with the science program, we did 
not have set times for [science], so this year, we definitely budgeted in that time. We 
said, “This is when we’re doing science, and that’s it. ’’ 

In interviews, teachers from both cohorts were asked to reflect on how their teaching changed 
since their school’s involvement with SIE began. The overarching sentiment was that teachers 
are both teaching more science and using more hands-on activities to do so. Moreover, many 
teachers indicated that science has taken on a greater a priority at their schools. The following 
responses were typical: 

Well, we didn’t teach any science [before SIE[. So, we went from nothing to something. 
So, I guess it drastically changed. 

Before, I would say we taught anywhere from fifteen minutes to thirty minutes a day, 
maybe two times a week, and then once we did the [module] we devoted about an hour a 
day, two to three times a week with the program. 
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[My instruction ] has changed. It has become more hands-on. Before we would 
complain that we thought that our science was basically worksheets and that was it. And 
now, from learning SIE’s hands-on approach, we are able to use that with other themes 
that we do in our science program. 

17 

The module use survey also asked teachers about the extent to which they used the readings 
and extension activities that came with the modules. As can be seen in Table 18, there was a 
great deal of variability in the use of these supplemental materials. Given that a number of the 
readings contain important concepts that are not covered elsewhere in the modules, and the push 
to integrate science and literacy, it is surprising that the readings are not being utilized more. 

The program may want to consider ways to emphasize the importance of the readings to teachers 
in the professional development. 



Table 18 



Use of Supplemental Module Materials 





Percent of Classes 


Readings (N = 386) 




None 


4 


About one-quarter 


21 


About one-half 


22 


About three-quarters 


18 


All/nearly all 


35 


Extension Activities (N = 499) 




None 


31 


A few 


47 


Many 


15 


All/nearly all 


7 



Student Opportunity to Learn 

To gain insight into how the modules and the associated professional development were being 
translated into classrooms, members of the evaluation team observed a sample of 48 participants 
teaching a lesson from a module. The classroom observations focused on the quality of the 
instruction in terms of opportunity for students to deepen their understanding of the intended 
content. Interviews after the observations were used to gather information on the teachers’ 
experiences with the module and SIE professional development. 

Sixteen Cohort 1 teachers were observed in the Fall of 2007 teaching their first SIE module for a 
second time. HRI selected eight schools for the Fall observations; three of these had been visited 
in Year One and five were randomly selected for Year Two. Two teachers were selected for 
observation within each school. Observations were also conducted in the Spring of 2008: 16 
Cohort 1 teachers implementing their second SIE module for the first time, and 16 Cohort 2 
teachers using a module for the first time. Schools and teachers were randomly selected for 



If applicable; some modules did not include readings. 
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these observations. For all three sets of observations, the evaluation focused on modules that 
were also part of the student achievement study described later in this report. 

The framework HRI used for assessing student opportunity to learn is based on research on how 
people learn. The framework specifies that learning is facilitated by a supportive classroom 
climate that encourages active participation of all students. Research also suggests that science 
instruction is most effective when instructional activities are well-aligned with the learning goal, 
engage students intellectually (i.e., “minds-on” rather than just “hands-on”) with important and 
accurate content at an appropriate level of difficulty, reflect the nature of science (e.g., requiring 
conclusions to be supported by data/evidence), and help students make sense of the important 
ideas. Following is an analysis of the observed lessons on each element of this framework, as 
well as an analysis of factors that may have affected student opportunity to learn. 

Extent To Which the Classroom Culture/Learning Environment Facilitated Student 
Opportunity To Learn 

A classroom culture that is respectful of students, encourages students to participate and share 
ideas, and maintains high expectations for all students aids student learning. The vast majority 
of the observed lessons had supportive learning environments characterized by a collegial 
atmosphere that encouraged student participation. For example, an observer reported: 

In a 6* grade lesson from the Experiments with Plants module, students worked cooperatively to determine 
what variable they were going to test, and what their research question would be, in planning a germination 
experiment. The classroom environment established by the teacher was warm and supportive. It was 
evident that students felt comfortable sharing their ideas and discussing their thoughts from the large 
number of students who participated in the class discussions. Students enthusiastically answered questions 
from the teacher and were eager to share their ideas. For example, during one class exchange, students 
readily offered their ideas as to which factors they could test in the experiment they were designing. 

In a handful of the observed lessons, classroom management issues detracted from the classroom 
culture. For example, in one observed lesson, while the teacher worked with an individual 
student the remaining students were off-task and disruptive; one student used the materials for 
the experiment to drum on the table and another student was shooting blocks into a bucket. 
Because the teacher had difficulty managing the learning environment, very little progress was 
made on the lesson goals 

In a few other lessons, observers noted that the teacher seemed to favor some students which 
resulted in different opportunities to learn for different students. For example, during the large 
group discussion at the end of a Levers and Pulleys lesson, the teacher called on the same subset 
of students to come to the board and answer questions. Students that were not included in the 
discussion tended to disengage from the lesson. 

Although it is beyond the scope of the SIE program to teach basic classroom management skills, 
the program may want to consider options to provide extra assistance to teachers who struggle 
with managing a hands-on learning environment. 

Extent To Which the Disciplinary Content Experienced by Students Was Accurate 
Observers examined the accuracy of the disciplinary content students experienced, including the 
content presented by the teacher, and the conclusions drawn from data/evidence. Of the 48 
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observed lessons, 41 had a learning goal related to science content (7 lessons focused solely on 
process skills), and the science content was accurate in the majority of these. However, there 
were content errors present in a number of the lessons. Some of the inaccuracies were relatively 
minor in nature. 

However, in eight of the observed lessons the content errors were quite problematic. Sometimes, 
the teacher introduced inaccurate information to students. For example, in a Rocks and Minerals 
lesson, students were learning about the different types of crystalline structures of minerals (i.e., 
the regular internal geometric pattern present in most minerals regardless of their outward 
shape). Although the teacher accurately defined fracture (the shape and texture of the surface 
formed when a mineral is broken) and cleavage (a fracture in which the mineral breaks along flat 
planes) to the class, she provided inaccurate information in examples supplementing the lesson. 
The teacher indicated that the fracture of minerals was similar to what occurred when a human 
bone was broken: “Think of a bone fracture. It isn’t a straight line but jagged. A fracture does 
not break in a particular shape.” In fact, the pattern of fracture is one of the features used to 
identify the type of mineral. The use of this analogy, although an attempt to relate an abstract 
science concept to something students were familiar with, likely interfered with students’ 
understanding the concepts in the lesson. 

In other cases, the teacher omitted important information and as a result may have led to a 
misconception. For example in a 6* grade lesson from Experiments with Plants, the teacher 
presented and elicited partially inaccurate information about the life cycle of a flowering plant. 
The teacher had students call out the steps of the life cycle and drew them on the chalkboard. 

The students omitted one of the steps and the teacher did not correct them or fill in the step. In 
addition, the teacher did not make the distinction that the life cycle they were discussing was 
only for flowering plants and did not apply to all plants. 

Extent To Which the Nature of Science Was Accurately Portrayed 
Understanding the nature of science as a discipline is an important aspect of learning about 
science. Students should understand that scientific knowledge is based upon the drawing of 
conclusions on data/evidence and is not just the memorization of facts. Of course, in order for 
students to draw conclusions from data they collect, the data must be collected carefully and 
systematically. Thus, observers also examined lessons for how scientific knowledge was 
portrayed and whether data collection would facilitate the drawing of appropriate conclusions. 

In 28 of the 48 observed lessons, variables were controlled during experimentation and data were 
recorded carefully. For example: 

In a lesson from the Environments module, 5* grade students investigated how their animals (pill bugs and 
darkling beetles) responded to moisture. After reviewing what living and non-living environmental factors 
are, the teacher demonstrated the experimental design for the lesson and emphasized the importance of 
keeping all but one variable (in this case the moisture of the soil) the same. Working in groups, students 
conducted the investigation by placing one cup of dry soil, one cup of moist soil, and one cup of wet soil in 
separate piles in the animal runways the students had previously made. Students carefully measured the 
cups of soil to ensure they were equal and made sure they didn’t touch. Students then recorded their 
experimental setup in their science notebooks. After placing their animals in the runway, students wrote 
down their observations of the animals for the next few minutes, then again at the end of the period. They 
would make a final observation of their animals the following day. 
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In contrast, students were either not controlling variables or not making careful measurements in 
20 of the observed lessons. In some cases, the teacher appeared unaware of the consequences of 
poor data collection. In a lesson from Mixtures and Solutions, the students were supposed to 
learn the concept of solubility, including that solubility varies depending on the materials. 
Although the teacher stressed the importance of keeping accurate records of their experimental 
procedure, when students made data collection errors, the teacher was not concerned. As the 
observer described: 

Several groups were not careful in handling the experimental solution and spilled some of their solution. 
Neither the students nor the teacher seemed to recognize that this was a potential source of error in the 
experiment. After observing one group’s spill, the teacher told the students, “Don’t worry about it and just 
keep going.” At no point did the teacher revisit this situation and indicate how it may have affected the 
results. 

Finally, observers noted in many of the lessons that a lot of data collection occurred, but very 
little drawing of conclusions from that data was present. Although careful data collection is a 
pre-requisite step, the critical component of drawing and critiquing conclusions based upon those 
data was absent in many of the lessons. 

Extent To Which All Students Were Intellectually Engaged with the Targeted Ideas in the 
Lesson 

High quality science lessons engage students intellectually with the targeted science ideas. In 
assessing this aspect of instruction, observers examined the extent to which students were 
thinking and talking about the intended science content. 

Observers found evidence of intellectual engagement in 34 of the 48 lessons. In 20 of these 
lessons, the observers indicated that students were highly focused on the lesson objectives. 
Typically, these lessons were both structured and facilitated in ways that fostered intellectual 
engagement. For example, both the requirements of the activity and the teacher’s questions 
engaged 3’^‘* graders in a lesson from the Rocks and Minerals module: 

In small groups, students were actively engaged in using their senses to make observations of rock samples. 
Students were also recording their observations in their science notebooks. The following exchanges are 
typical examples of how the students described the properties of their rocks. 

Student 1: It’s bumpy. All of it’s bumpy. 

Student 2: It’s shiny, with black stripes. 

Student 3: There is no sound, but it feels rough. 

Student 4; It’s shiny. 

Further, the teacher’s questions helped focus students on identifying distinguishing features of the rock 
samples: 

Teacher: You can’t just say something smells horrible. . .What does it smell like? 

Student 1 : It smells nasty, like trash. 

Teacher: How about the weight of the rock? (She models feeling the weight of the rock by gently tossing it 
and letting it fall into her hand, which the students in several groups watched and proceeded to do 
themselves.) 
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The teacher’ s questions focused students on making detailed descriptions, which at the end of the lesson 
helped them to make sense of what a “property” is. 



In 14 lessons, the observers indicated that the level of intellectual engagement was mixed. In 
some cases, students were focused on the intended content during parts of the lesson, but not 
throughout. In other cases, students were focused on only a subset of the intended goals. For 
example: 

In a lesson from the Ecosystems module, students were intended to deepen their understanding of what an 
ecosystem is by comparing and contrasting aquatic and terrestrial ecosystems. Students made observations 
of their terrariums and aquariums with hand lenses, and recorded their observations in their science 
notebooks. The class also engaged in a discussion about the plants and animals they observed, recording 
similarities and differences on a large Venn diagram in the front of the room. However, the lesson never 
had students think about how these examples related to ecosystems. 

In the remaining 14 observed lessons, students were not focused at all on the intended content, 
often because students were concentrating on the process of completing an activity rather than 
the science content. For example: 

Students in a 3^“^ grade class working in the Chemical Tests module were adding water to five unknown 
substances to learn that some substances dissolve in water and others do not. This lesson was implemented 
in a hands-on manner, but was not minds-on for the students. During the activity, student conversations 
focused on what numbers to record, but not what the numbers meant; many students did not appear to 
understand what they were doing and why they were doing it. In addition, the teacher’s questions during 
the whole class discussion focused on the procedure of the activity and not what students were finding. 

Extent To Which Students Were Given the Opportunity To “Make-Sense” of the Targeted 
Ideas 

The framework used to examine the sample lessons asserts that high-quality instruction involves 
the teacher and/or instructional materials helping students make meaningful and relevant 
connections between their experiences and key science concepts (i.e., “making-sense” of the 
targeted ideas). 

Twenty-seven of the observed lessons contained elements of sense-making, though only 12 were 
judged to have adequate opportunities for students to make sense of the targeted science 
concepts. The common element in lessons with sense-making activities was the teacher helping 
students make connections between the instructional activities and science concepts, typically 
through a whole class discussion at the end of the lesson. For example, in a grade lesson from 
the Environments module: 

The focus of the lesson was for students to learn that carbon dioxide produced by aquatic organisms 
increases the acidity of the water. During the lesson, students set up an investigation with three cups, each 
containing lOOmL of water. In Cup 1 students added a fish; in Cup 2 they added a sprig of Elodea (aquatic 
plant); nothing was added to Cup 3. Six drops of the acid indicator bromothymol blue (BTB) were added 
to each cup. The teacher led a group discussion to help students make the connection between the activity 
the targeted concept. The discussion included: 

Teacher: Okay, why do you think the fish water is turning yellow? Why does it have acid in it? 

Student 1: Oil on (the fish’s) body. 

Student 2: Fish went to the bathroom. 

Teacher: It didn’t go that much (laughing). 
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Student 3: Exhaling CO 2 ? 

Teacher: We are going to test that theory. Blow into the water with this straw. [Student blows into the cup 
with plain water and BTB.] Stop, what color is your water? 

Student 3: It turned yellow! 

Teacher: Why? 

Class: CO 2 ! 

Teacher: When CO 2 and water combine they form an acid called carbonic acid. So, what is BTB testing 
for? 

Student 4: Chemicals in water. 

Teacher: Something specific . . . 

Student 1 : Acid in the water. 

Teacher: If there is acid in the water, what color will the BTB turn? 

Class: Yellow. 

Fifteen lessons had some, but rather limited opportunities for students to make sense of the 
science ideas and 16 lessons provided little, if any, opportunity for students to make sense of the 
targeted ideas. In these lessons, there tended to be no structured time for students to make sense 
of the targeted ideas. Sometimes the teacher focused the wrap-up discussion on the process of 
the activity instead of interpreting the data. In other cases, the teacher accepted the conclusion of 
a single student even though the supporting evidence was incorrect: 

In a Levers and Pulleys lesson, 5* grade students were examining what happened to the effort required to 
lift a load when the effort force is moved farther and farther from the fulcrum in a lever system. The 
students were shown two lever systems and asked to predict which lever would make the load easier to lift 
and explain why. The lever systems were the same except that one had the effort force farther from the 
fulcrum than the other. Students then tested their predictions and the teacher asked the class what they 
found. One student gave the correct answer, but indicated that the reason was that the fulcrum was closer 
to the load. The teacher agreed and told the rest of the class to write the “correct” answer in their 
notebooks. 

In five of the observed lessons, although no sense-making occurred, the teacher indicated in the 
post-observation interview that the class would discuss their data the next day. 

Fidelity of Implementation 

Twenty of the observed lessons were implemented with a high degree of fidelity. Of these, 15 
provided a great deal of opportunity for students to learn the targeted ideas; 5 lessons were 
implemented with fidelity but judged not to provide sufficient opportunity to learn, reflecting 
gaps in the instructional materials. For example, a lesson from the Motion and Design module 
has students investigating the relationship between the force applied to their vehicle and the 
vehicle’s motion. Students change the number of washers on the end of a string attached to a 
vehicle, hanging the washers over the edge of the table, which applies a forward force on the 
vehicle. The activity asked students to make observations about the speed of the vehicle in each 
trial. As the vehicle started each trial from rest, the final speed of the vehicle was greater when 
more washers were added to the string because the vehicle undergoes greater acceleration. 
However, this connection between final speed and acceleration is not made explicit in the 
module. In a 4* grade class observed implementing this activity, the observer noted that the 
teacher did not help students make the connection between the data they were collecting and the 
targeted science concept even though the teacher had followed the lesson exactly as it was 
described in the teacher’s manual. 
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Teachers made substantive modifications in the other 28 observed lessons. In these cases, 
observers indicated that the modifications reduced student opportunity to learn. Interestingly, 
lessons intended to focus on disciplinary content were more likely be modified by the teachers 
than lessons that focused on teaching process skills. The most common change was the omission 
of the wrap-up discussion and associated questions outlined in the instructional materials, 
reducing the sense-making opportunities in the lessons. For example, in a lesson from the 
Human Body module, students examined animal skeletons found in owl pellets and compared the 
skeleton of a rodent to the skeleton of a human. The intended goal of the lesson was for students 
to see that bones that serve similar functions have similar shapes. During the lesson, the teacher 
has students find the bones in the owl pellets and reconstruct a rodent skeleton as directed in the 
module. However, the teacher cut out the portions of the activity that asked students to compare 
the rodent skeleton to the human skeleton and consider the relationship between bone shape and 
function. 

Another type of modification that reduced the emphasis on the science content was more subtle. 
Some lessons that included investigations in which students were collecting data that could be 
used to develop a science concept were implemented in a way that focused students on the data 
collection process rather than making meaning from the data. For example, an observed Electric 
Circuits lesson was intended to teach students that electric current can produce light, heat, and 
magnetism as evidence for energy being transformed. During the lesson, the teacher focused 
students solely on properly following the procedures. In fact, the teacher graded students on 
setting up the experiment properly, regardless of whether or not they collected meaningful data. 
The idea that electric energy can be converted into other forms of energy was never brought forth 
in the lesson. 

Overall Extent To Which the Lessons Provided Students with an Opportunity To Learn the 
Targeted Ideas 

All of the observed lessons included some elements of effective practice, though only a few 
lessons managed to “put it all together.” Common strengths of the lessons included involvement 
of students with key content as well as collegial interactions among students and respectful 
contact between the students and the teacher. Common areas of weakness included the extent of 
student intellectual engagement with, and sense-making of, the content. Students engaged in 
many hands-on activities, but the amount of intellectual rigor required of the students in most 
lessons was generally low. No major differences were found between lessons implemented by 
Cohort 1 and Cohort 2 teachers; as most of the observations of Cohort 1 teachers occurred prior 
to the module enrichment sessions, these data indicate that simply teaching a module for a 
second time is unlikely to have a major impact on the quality of implementation. 

Relationship Between SIE Professional Development and Teachers’ Perceptions of 
Preparedness To Implement the Module 

Teachers’ confidence with the content in the modules and familiarity with the instructional 
materials can have a great impact on the quality of implementation. Teachers were asked about 
each of these areas during interviews following the observed lessons. 

Thirty-seven of the 48 observed teachers indicated they were comfortable with the content in the 
module; most attributed this comfort to the SIE professional development though some indicated 
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that they were comfortable with the content even before participating in SIE. The remaining 1 1 
teachers indicated feeling uncomfortable with the content in their module. As a few said: 

[The SIE professional development] didn ’t do much for me. At first, I was a little 
overwhelmed with it, but I took a lot of time on my own to explore and to understand it 
online a little bit, but mostly from the teacher manual. 

We went over every lesson, and we got to do what the children do. But that didn ’t really 
help me learn the science. I could have used a little more background knowledge on the 
concepts. It was a little new at first. 

Nearly all of the observed teachers felt very confident to use the module in the classroom. Most 
attributed this confidence to the SIE professional development, saying that going through the 
activities in the training was very helpful. In the words of a few teachers: 

Going through it like the kids would, helped me see what to expect. 

[The training] gave me ideas on how to go about teaching, you know what kind of 
strategies to use with them and it really worked. 

The training was wonderful. We had a woman that had actually taught the module 
herself and she gave us tips on what actually worked practically in the classroom verses 
maybe what the book says to do. 

Overall, most of the observed teachers indicated being well prepared to implement the module, 
both in terms of familiarity with the instructional activities and the science content. However, 
teacher perceptions do not appear to correlate with the quality of the observed lessons, indicating 
that teachers may not yet have a strong vision of high quality science instruction or how to use 
the module to enact that vision. 
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Impact of SIE on Students 



As part of the evaluation of the SIE program, HRI, with the assistance of the PDE, designed and 
implemented a study to examine the impact of SIE professional development and teachers’ 
implementation of the science modules on student achievement in science. The study utilized a 
treatment/comparison group pre-test/post-test design to compare assessment results for students 
receiving instruction on a topic using the SIE-provided modules to those not receiving 
instruction on that topic. 

In Year Two of the SIE program, assessments were administered in both the Eall of 2007 and the 
Spring of 2008. In the Eall, assessments related to six modules were administered by Cohort 1 
schools prior to and just after their implementation of their first module. In the Spring, 13 
assessment scales were administered: Cohort 1 schools administered a second set of assessments 
for their second module and Cohort 2 schools administered assessments for their first module. 
Eull results from the analyses of these data are reported separately ; highlights of the major 
findings follow. 

Do students of teachers who participated in SIE professional development and who 
implemented the SIE-provided module on a topic exhibit greater achievement in science after 
instruction than students of teachers who participated in SIE professional development and 
who implemented a module in a different topic? 

In all but one instance, the treatment and comparison groups of students were equivalent 
initially; for the exception, the comparison group outperformed the treatment group on the pre- 
test. On the post-test, controlling for pre-test scores and student demographics, treatment group 
students significantly outperformed comparison group students on 5 of the 6 assessment scales in 
the Eall of 2007 and on 1 1 of the 13 scales in the Spring of 2008. Effect sizes ranged from 0. 18 
to 1.16 standard deviations; most were at least 0.75 standard deviations. 

Are there gender or race/ethnicity differences in student achievement, and if so, does use of an 
SIE-provided module reduce these differences? 

The statistical models also examined whether achievement gaps existed by gender and 
race/ethnicity, and if there was adequate class-level variation, whether the use of the SIE- 
supported module changed the achievement gap. Because the number of students classifying 
themselves as any race/ethnic group other than White was small, these data were collapsed into 
two categories: White/Asian vs. non- Asian minority.*^ 

White/Asian students outperformed non- Asian minority students on each scale at each pre-test 
administration. All but two of these achievement gaps did not change from pre- to post-test. On 
the Levers and Pulleys scale in the Eall of 2007, although both White/Asian and non- Asian 



** Banilower, E.R., Weis, A.M., & Rosenberg, S.L. (2008). Examining the Impact of Science: It’s Elementary 
on Student Achievement; Analysis of Eall 2007 Data. Chapel Hill, NC; Horizon Research, Inc. 

Rosenberg, S.L., Weis, A.M., & Banilower, E.R. (2008). Examining the Impact of Science: It’s Elementary 
on Student Achievement; Analysis of Spring 2008 Data. Chapel Hill, NC; Horizon Research, Inc. 

Asian students typically outperform all other groups of students, and are often grouped with White students in 
these types of analyses. 
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minority students in treatment classes made large gains from pre- to post-test (compared to a 
minimal gain for students in the comparison classes), the rate of growth was greater for 
White/Asian students than for non- Asian minority students. A similar pattern was found for the 
Rocks and Minerals scale in the Spring of 2008. 

In terms of gender gaps, male students outperformed female students on 3 of the 6 assessment 
scales in the Fall of 2007 and 5 of the 13 scales in the Spring of 2008. In addition, females 
outperformed males on one pre-test scale in the Fall of 2007. None of these gaps changed from 
pre- to post-test. However, post-test gender gaps were found on four scales that did not have 
pre-test gender gaps. In three cases, females outperformed males on the post- test in both 
treatment and comparison classes. In one case, males outperformed females on the post-test in 
treatment classes (although both groups had large gains); neither males nor females had 
substantial gains in the comparison classes. 

Is there a relationship between extent of use of the SIE-provided modules and student 
achievement? 

HRI examined whether the extent of module use was related to student achievement. A module 
use survey administered at the time of the post-test asked teachers to indicate which lessons from 
the module they had implemented, the amount of instructional time devoted to the topic, and the 
extent to which that instructional time was based on the module vs. other instructional materials. 

HRI used a statistical technique called “cluster analysis” to classify classes into groups with 
distinct patterns of module use. On the 15 scales for which there was variation in module use, 
HRI found a significant positive relationship between extent of module use and post-test scores 
on 7; no relationship was found for the other 8. 

Is teachers’ instruction more effective the second time they implement a module than the first 
time they implemented it? 

By combining data collected in Year One and Fall of Year Two of SIE, it was possible to 
examine whether a “practice” effect exists. In other words, with no additional professional 
development, is instruction likely to be more effective when teachers are using a module for the 
second time? For this analysis, HRI selected only those teachers who used the same module in 
Year One and Year Two of the program. 

On each scale, pre-test scores were significantly lower the second time teachers were 
implementing their module. This result may be due to the fact that the pre-test (and post- test as 
well) assessment was administered in the Fall of Year Two compared to the Spring of Year One. 
The cognitive and reading ability of elementary age children can change substantially in a short 
period, and the several month difference in when the assessment was administered may be 
responsible for the initial differences in scores. 



Cluster analysis is an exploratory statistical technique that groups cases on a combination of variables such that 
people within a group are similar to each other and relatively distinct from other groups. HRI employed the SPSS 
two-step cluster analysis procedure for this analysis using the percent of the module covered and the total 
instructional time spent on the topic variables. Percent of instructional time based on the module was not used to 
form the groups in the cluster analysis because the clustering solution resulted in more distinct differences among 
the groups when this variable was excluded. 
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Although pre-test scores were significantly lower the second time teachers implemented the 
module, post-test scores were not significantly different between the two times for 5 of the 6 
scales (the exception being Electric Circuits), indicating that student growth was greater the 
second time teachers were implementing these modules. One possible explanation for these 
results is that teachers were able to use more of the module; another is that they spent more time 
on the topic. Although the sample size was too small to statistically examine whether extent of 
module use differs between first and second implementation, a descriptive analysis of the module 
use data revealed that teachers, on average, appeared to devote much more instructional time to 
the module during their second implementation than during their first implementation even 
though the two groups covered roughly the same number of lessons in the module. 

Did Cohort 2 have greater gains their first time using the modules (in 2007-08) than Cohort 1 
did their first time using the same modules (in 2006-07)? 

HRI analyzed the assessment data related to the six original modules that were implemented by 
both cohorts (Rocks and Minerals', Electric Circuits', Levers and Pulleys', Motion and Design', 
Mixtures and Solutions; and Variables) to examine how changes in the initial module training 
provided by the SIE program may have affected student learning. Data from the first two years 
of the SIE program were combined for these analyses, and only students of treatment group 
teachers were included. 

On each assessment scale related to these six modules, scores of students in the two cohorts were 
initially equivalent. Eurthermore, there were no significant differences in pre- to post-test score 
gains between students of teachers in Cohort 1 and students of teachers in Cohort 2 for any of the 
modules. Thus, it does not appear that the changes made to the initial module training between 
the first and second year of the program led to changes in student achievement as measured by 
the assessments used in this study. 

Is there a relationship between teacher participation in SIE Module Enrichment workshops 
and student achievement? 

As part of the program design, SIE is offering teachers a second round of professional 
development on their modules. In October 2007, SIE piloted these module enrichment 
workshops for four modules with a subset of teachers participating in the program. The four 
modules were Electric Circuits, Levers and Pulleys, Motion and Design, and Mixtures and 
Solutions. However, it is important to note that teachers received their modules at the beginning 
of September 2007, and many teachers were several weeks into teaching the module before 
participating in a content deepening workshop. Thus, the opportunity for the workshops to have 
an impact on student achievement was limited. 

Sixty-one teachers attended one of the pilot module enrichment workshops. However, not all of 
these teachers had participated in the initial training for the module; these teachers were not 
included in the analysis. To control for the possibility that the teachers attending the module 
enrichment workshops were not typical (i.e., they were less or more effective than most 
teachers), HRI decided to compare the Year One student achievement of teachers attending these 
sessions to those not attending. 
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There was no significant difference in pre-test scores by whether the teacher had attended a pilot 
module enrichment workshop, indicating initial equivalence of students of the two groups of 
teachers. In addition, there were no differences in Year One data between the two groups of 
teachers; in other words, teachers who attended the pilot module enrichment workshops in Year 
Two were not more or less effective than their colleagues in the year prior to attending one of 
these workshops. 

Statistically, there was no significant difference in Year Two post-test scores between students of 
teachers who did and did not attend one of the pilot module enrichment workshops. However, it 
is interesting to note that the statistical test, although not meeting the criteria for significance, 
was close to being significant (an exact p-value of 0.054; p-values < 0.050 are typically 
considered significant), with students of attendees scoring 0.17 standard deviations higher on the 
post-test than students of non-attendees (controlling for pre-test scores and student demographic 
characteristics). It is not possible to determine whether this apparent difference is simply noise 
due to a small sample size or a real difference that this study lacked the statistical power to 
detect. Data to be collected in Year Three of the program may allow for a more definitive 
exploration of the impact of these module enrichment workshops on student achievement. 
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Summary and Recommendations 



In Year Two, the SIE program can be credited with a number of accomplishments. The program 
has brought on board a second cohort of schools, in addition to increasing the services added to 
the first cohort. SIE has implemented Strategic Planning Institutes and Vision Conferences for 
Cohort 2 schools; provided three days of professional development to Cohort 2 teachers across 
the state; provided two days of professional development to Cohort 1 teachers on a second 
science module; provided two days of enrichment training to a subset of Cohort 1 teachers on 
their initial module; developed and delivered a leadership conference for teacher leaders; and 
delivered ready-to-use science modules to teachers participating in the program. 

Overall, teachers, principals, and district staff have reported positive experiences with the SIE 
program. The Strategic Planning Institutes provided an opportunity for school teams to become 
familiar with science education reform and gave them the rare opportunity to work together on a 
plan for improving their science program. The Vision Conferences provided schools the 
opportunity to broaden the base of stakeholder support for science education reform. 

Initial module training continues to be extremely well received by teachers and has generated a 
great deal of enthusiasm for teaching science from the modules, even among teachers who were 
skeptical of the program at the start. In addition, the professional development was successful at 
providing teachers with the knowledge and skills they needed to feel confident in implementing a 
science module for the first time. 

The SIE program has implemented, and continues to refine, module enrichment training to help 
teachers become more purposeful in their teaching from the modules. Given the relative 
inexperience of elementary teachers with science, providing them with all of the knowledge they 
need is a tall order. Data from a pilot study indicate that SIE is having an impact on teacher 
content knowledge, though many teachers still have considerable room for growth in this area. 

Classroom observations of teachers implementing the modules found that all of the observed 
lessons included some elements of effective practice, though only a few lessons managed to “put 
it all together.” Common strengths of the lessons included involvement of students with key 
content as well as collegial interactions among students and respectful contact between the 
students and the teacher. Common areas of weakness included the extent of student intellectual 
engagement with and sense-making of the content. Students engaged in many hands-on 
activities, but the amount of intellectual rigor required of the students in most lessons was 
generally low. Many teachers made modifications to the instructional materials, often in the 
form of deleting probing questions or wrap-up discussions, which likely hindered students’ 
opportunity to make sense of the target concepts. 

Analyses of student achievement data provide evidence that the SIE program is having a positive 
effect on student learning in science. On 5 of the 6 assessment scales administered in the Eall of 
2007, and on 1 1 of the 13 scales administered in the Spring of 2008, students receiving 
instruction using the SIE-provided module related to the scale scored significantly higher than 
students not receiving instruction from that module, controlling for pre-test scores and student 
demographics. 
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Although the SIE program has accomplished a great deal in its second year, as is the case with 
most programs in their early stages, a substantial amount of work remains to be done. In the 
spirit of a critical friend, HRI offers the following recommendations to assist the program in 
reflection and planning for the future. 

> Consider ways to support districts, schools, and teachers in developing articulated 
curricula that incorporate the SIE-provided modules and that also ensure that all state 
science standards will be addressed. 

Although the majority of districts participating in SIE report having curriculum guides to 
direct teachers in the selection of science topics to teach (most of which were developed prior 
to SIE), evaluation data indicate that the enacted curriculum may not align well with 
Pennsylvania science standards. This lack of alignment is likely due to at least two factors. 
Eirst, because of the emphasis on reading/language arts and mathematics due to high-stakes 
testing, science (and other non-tested subjects) has been deemphasized in schools. Second, 
elementary teachers often do not feel well prepared to teach science, consequently 
minimizing the amount of science they teach. Teachers also tend to cover the topics in which 
they feel comfortable and skip the others. 

Although the addition of a state science assessment has encouraged schools to include more 
science in their instructional programs, the fact that science is not counted towards Adequate 
Yearly Progress as part of No Child Eeft Behind may temper the impact of the science PSSA 
in this regard. Schools may find information on how they can reach literacy goals through 
the teaching of science helpful, as well as other ideas on how to make time for science 
instruction. 

> Continue to refine the initial module training provided to teachers, including 
strengthening teachers’ vision of effective inquiry-based science instruction. 

The professional development sessions HRI observed varied widely in regard to their focus 
on the science content and the content storyline in the modules. In some sessions, facilitators 
consistently made explicit connections between module activities and content goals, but in 
other sessions they did not. Thus, the program should consider building a structure into the 
training (i.e., some sense-making activity for each lesson in a module that all facilitators 
would be expected to do) to help ensure that all participants recognize the intended content in 
each lesson. 

Teachers would also benefit from understanding how the different science concepts in the 
module fit together and build towards the big ideas in science. Thus, the program should 
consider increasing the focus on the content storyline of a module in the initial module 
training. The program may also want to consider providing teachers with a hand-out that 
summarizes the concepts covered by the module and their interrelationships, as well as how 
each of the concepts is developed through the various module activities. Such a document 
would help teachers see how each activity in the module adds to student learning, reducing 
the likelihood that teachers pressed for time will skip key activities. 
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In addition, for teachers to effectively use inquiry-based teaching methods, they need to have 
a clear vision of what inquiry instruction is and how they can use the module in an inquiry- 
based manner. In the three-day initial module trainings HRI observed, the concept of inquiry 
was discussed during the first day’s overview, but no clear vision of inquiry ever emerged. 
Understanding that the process of science and the nature of scientific knowledge rest upon 
drawing conclusions from evidence is key to effectively using inquiry in the classroom. 
Without this understanding, as well as a solid understanding of the concepts they are 
expected to teach, teachers are unlikely to be able to move from mechanical to purposeful 
implementation of a module 

> Classroom observations make it clear that most teachers need additional support to 
attain purposeful use of a module. The program wisely includes sustained professional 
development for teachers to help them develop the additional knowledge needed. The 
program should continue to refine these workshops. 

The module enrichment workshops have had a number of benefits for the attending teachers, 
including greater feelings of preparedness to implement the module and deeper 
understanding of the science content in the module. However, as with any new undertaking, 
these workshops would benefit from some refining. 

Although the program has incorporated a number of activities in the module enrichment 
workshops that are rooted in what is known about effective professional development (e.g., 
examining student work, considering formative assessment strategies), many of the 
participating teachers were unable to fully engage with the activities. This lack of 
engagement was likely due to teachers not yet having a solid enough understanding of the 
science content of the module. Making the science content in the module more explicit in the 
initial module trainings will likely improve teachers’ readiness for these types of activities. 
Teachers would also benefit from a somewhat deeper understanding of the content so they 
could effectively address questions students may raise or recognize and be able to redirect 
students when they begin traversing unproductive paths of inquiry. 

In addition, it is important for the program to consider ways to tie the teachers’ experiences 
in the module enrichment workshop more directly to the module. Because teachers are so 
pressed for time, they often need to see the direct application of what they are learning to 
their classroom practice. Making sure all of the workshop activities have a direct tie-in to the 
module, and helping teachers see these connections, will increase the likelihood that they will 
engage with these activities. 

> As the program continues to expand and bring aboard new schools and new 
professional development providers, it should consider creating artifacts that capture 
key elements of the program and its work. 

Creating a content storyline document that highlights the science concepts covered by a 
module, demonstrates the interrelationships among these concepts, and describes how each 
concept is developed via the module activities would likely benefit the program in several 
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ways.^^ First, this document would help guide the professional development around the 
module, ensuring that all of the important ideas are addressed. Second, this document would 
aid in the training of new professional development providers, ensuring consistency across 
the different sessions. Third, providing this document to teachers would give them a 
resource that they can refer back to during module implementation (which is often several 
weeks or months after the professional development), and help teachers move along the 
trajectory toward purposeful implementation of the module. 



Over time, as resources permit and for topics in which the research exists, a content storyline could be expanded 
to include a summary of the common student misconceptions related to the topic as well as tips for addressing them. 
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District Curriculum Alignment Questionnaire and Summary of Data 
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Fall 2007 District Curriculum Alignment Questionnaire 



http://www.horizon-research.com/sie/dist_curric_fall07/daq.php 



District Curriculum Alignment Questionnaire 

Fall 2007 



The Pennsylvania Department of Education (PDE) has requested that each district 
participating in the Science: It's Elementary program complete this questionnaire. 
The responses will help PDE decide what support to offer districts in future years, so 
please be candid. If you have any questions about this survey, please email Camille 
Warren at sie_eval@horizon-research.com or call (toll-free) 877-297-6829. 



1. Does the district have a curriculum guide for science in grades K-6? 

r Yes 
r No 

If you answered "no" to question #1 above, click here to skip to question #10 . 

2. What is this guide called in your district? 



3. Was a process used to align the district K-6 science curriculum guide with 
the Pennsylvania Academic Standards for Science and Technology and 
Environment and Ecology? 

I - select one - 



If you answered "no" to question #3 above, click here to skip to question #7 . 

4. When was the most recent alignment process completed? 

I - select one - 



5. Who participated in the alignment process? (Select all that apply.) 
r~ Superintendent 

r~ Assistant superintendent 
r~ District science specialist/supervisor 
r~ Elementary school administrators 
r~ Elementary school teachers 
r~ Middle school science teachers 
r~ High school science teachers 
r~ University science educators 
r~ University/business/industry scientists 
I” Other (please specify): | 

6. Briefly describe the process that was used to conduct the alignment: 
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7. What are the functions of the district science curriculum guide? (Select all that 
apply.) 

r~ To specify the grade(s) at which each Pennsylvania standard should be taught 
r To help the district select instructional materials 

r To help teachers decide what topics to cover/instructional materials to use 
r other (please specify): | 

8. How was the district science curriculum guide for K-6 science communicated 
to teachers when it was developed? (Select all that apply.) 

r Each teacher responsible for teaching science in these grades received a copy of 
the district science curriculum. 

I The district provided staff development on the district science curriculum. 

I The district science curriculum was implicitly communicated through the selection 
of instructional materials, 
r other (please specify): | 

9. How is the district science curriculum guide for K-6 science communicated to 
teachers that have joined the district since it was developed? (Select all that 
apply.) 

r~ New teachers receive a copy of the district science curriculum. 

r The district provides staff development to new teachers on the district science 
curriculum. 

r The district science curriculum is implicitly communicated through the 
instructional materials, 
r other (please specify): | 

10. Prior to participating in Science: It's Elementary, approximately what 
proportion of K-6 science instructional time in your district was based on 
each of the following types of instructional materials? (Responses must add to 
100 %.) 

Textbook- based instructional materials 

Module/kit-based instructional materials 

Other (e.g., trade books, district/teacher-developed materials) 
instructional materials 

11. Which of the following best describes your district's policy regarding the 
selection of instructional materials for K-6 science? (Select one.) 

r a. All schools in the district are required to use the same instructional 
materials. 

C b. The district provides a list of materials from which individual schools may 
select. 



- proportion - 

- proportion - 

- proportion - 
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Fall 2007 District Curriculum Alignment Questionnaire 



http://www.horizon-research.com/sie/dist_curric_fall07/daq.php 



C c. Individual schools select instructional materials without constraints by the 
district. 

C d. Individual teachers select the instructional materials they want to use. 

If you selected "c" or "d" in question #11 above, click here to skip to question 
#15 . 

12. When were the current instructional materials for K-6 science adopted? 

I -- select one -- 



13. Who was involved in the selection of these materials? (Select all that apply.) 

1“ Superintendent 

r Assistant superintendent 
r District science specialist/supervisor 
r~ Elementary school administrators 
n Elementary school teachers 
n Middle school science teachers 
1“ High school science teachers 
r~ University science educators 
1“ University/business/industry scientists 
r other (please specify): | 

14. Please describe the process by which these materials were selected, including 
how the alignment between the materials and the district science curriculum 
and/ or the Pennsylvania Standards was assessed. 



15. About how much professional development did a typical teacher receive on 
these instructional materials? (Select one.) 

C None 

r 6 or fewer hours 
C Between 7 and 12 hours 
C Between 13 and 18 hours 
r Between 19 and 24 hours 
C 25 or more hours 

If you responded "none" to question #15 above, click here to skip to question 
#17. 



16. Who provided the professional development? (Select all that apply.) 



3 of 4 



7/8/2008 3:44 PM 



Fall 2007 District Curriculum Alignment Questionnaire 
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|~ The publisher 

r~ District science specialist/supervisor 
n Other district personnel (please specify): | 
r ASSET, Inc. 
n other (please specify): | 

17. When will the next adoption for K-6 science instructional materials take 
place? (Select one.) 
r 2007 

r 2008 
r 2009 

r 2010 
r 2011 

other, please specify: 



PDE has also asked that we interview a subset of the people responding to this 
survey. Please provide the following information so that we may contact you if 
you are selected for an interview. 
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SIE District Curricuium Aiignment Questionnaire - Fall 2007 



Number of valid questionnaires 





N 


Total Participants 


48 



Q1 





Total 


No 


Yes 


Valid N 


Missing N 


Percent 


Percent 


ql: Does the district have a curriculum guide for science in grades 
K-6? 


48 


0 


31 


69 



Q3 





Total 


No 


Yes 


Valid N 


Missing N 


Percent 


Percent 


q3: Was a process used to align the district K-6 science 
curriculum guide with the Pennsylvania Academic 
Standards for Science and Technology and Environment 
and Ecology? 


33 


0 


15 


85 



Table includes only those schools that have a curriculum guide for science in grades K-6 (Ql). 



Horizon Research, Inc. 































SIE District Curricuium Aiignment Questionnaire - Faii 2007 



Q4 





Total 


2001 or 
earlier 


2002 


2003 


2004 


2005 


2006 


2007 


Valid N 


Missing N 


Percent 


Percent 


Percent 


Percent 


Percent 


Percent 


Percent 


q4: When was the most recent alignment process 
completed? 


28 


0 


18 


11 


11 


4 


7 


32 


18 



Table includes only those schools that used a process to align their district K-6 science curriculum guide with the state standards. (Q3) 





Total 


No 


Yes 


Who participated in the alignment 
process? 


Valid N 


Missing N 


Percent 


Percent 


q5a: Superintendent 


28 


0 


86 


14 


q5b: Assistant superintendent 


28 


0 


75 


25 


q5c: District science specialist/supervisor 


28 


0 


71 


29 


q5d: Elementary school administrators 


28 


0 


32 


68 


q5e: Elementary school teachers 


28 


0 


7 


93 


q5f: Middle school science teachers 


28 


0 


39 


61 


q5g: High school science teachers 


28 


0 


54 


46 


q5h: University science educators 


28 


0 


89 


11 


q5i: University/business/industry scientists 


28 


0 


100 


0 


q5j: Other (please specify): 


28 


0 


68 


32 




Table includes only those schools that used a process to align their district K-6 science curriculum guide with the state standards (Q3). 
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SIE District Curriculum Alignment Questionnaire - Fall 2007 

Q7 



What are the functions of the district science curriculum guide? 


Total 


No 


Yes 


Valid N 


Missing N 


Percent 


Percent 


q7a: To specify the grade(s) at which each PA standard should be taught 


33 


0 


30 


70 


q7b: To help the district select instructional materials 


33 


0 


52 


48 


q7c: To help teachers decide what topics to cover/instructional materials to 
use 


33 


0 


15 


85 


q7d: Other (please specify): 


33 


0 


73 


27 



Table includes only those schools that have a curriculum guide for science in grades K-6 (Ql). 

Q8 



How was the district science curriculum guide for K-6 science 
communicated to teachers when it was developed? 


Total 


No 


Yes 


Valid N 


Missing N 


Percent 


Percent 


q8a: Each teacher responsible for teaching science in these grades received a 
copy of the district science curriculum. 


33 


0 


18 


82 


q8b: The district provided staff development on the district science 
curriculum. 


33 


0 


61 


39 


q8c: The district science curriculum was implicitly communicated through 
the selection of instructional materials. 


33 


0 


64 


36 


q8d: Other (please specify): 


33 


0 


76 


24 



Table includes only those schools that have a curriculum guide for science in grades K-6 (Ql). 
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SIE District Curricuium Aiignment Questionnaire - Faii 2007 



Q9 



How is the district science curriculum guide for K-6 science 
communicated to teachers that have joined the district since it 
was developed? 


Total 


No 


Yes 


Valid N 


Missing N 


Percent 


Percent 


q9a: New teachers receive a copy of the district science 
curriculum. 


33 


0 


30 


70 


q9b: The district provides staff development to new teachers on 
the district science curriculum. 


33 


0 


67 


33 


q9c: The district science curriculum is implicitly communicated 
through the instructional materials. 


33 


0 


58 


42 


q9d: Other (please specify): 


33 


0 


73 


27 



Table includes only those schools that have a curriculum guide for science in grades K-6 (Ql). 



Qio 



Prior to participating in Science; It’s Elementary, 
approximately what proportion of K-6 science 
instructional time in your district was based on each of 
the following types of instructional materials? 


Total 


0 


10 


20 


30 


40 


50 


60 


70 


80 


90 


100 


Valid N 


Missing N 


Percent 


Percent 


Percent 


Percent 


Percent 


Percent 


Percent 


Percent 


Percent 


Percent 


Percent 


qlOa; Textbook-based instructional materials 


46 


2 


9 


11 


2 


7 


7 


13 


13 


4 


15 


17 


2 


qlOb: Module/kit-based instructional materials 


44 


4 


25 


27 


9 


7 


0 


7 


7 


5 


5 


7 


2 


qlOc; Other (e.g., trade books, 
district/teacher-developed materials) instructional 
materials 


43 


5 


7 


37 


19 


19 


5 


7 


2 


0 


0 


0 


5 
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SIE District Curriculum Alignment Questionnaire - Fall 2007 



Qll 





Total 


All schools in 
the district are 
required to use 
the same 
instructional 
materials. 


The district 
provides a 
list of 
materials 
from which 
individual 
schools 
may select. 


Individual 
schools select 
instructional 
materials 
without 
constraints by 
tbe district. 


Individual 
teachers select 
the 

instructional 
materials they 
want to use. 


Valid N 


Missing N 


Percent 


Percent 


Percent 


Percent 


qll: Which of the following best describes your district’s 
policy regarding the selection of instructional materials for K-6 
science? 


48 


0 


56 


19 


13 


13 



Q12 





Total 


2001 or 
earlier 


2002 


2003 


2004 


2005 


2006 


2007 


Valid N 


Missing N 


Percent 


Percent 


Percent 


Percent 


Percent 


Percent 


Percent 


ql2: When were the current instructional materials for K-6 science 
adopted? 


36 


0 


33 


14 


6 


6 


17 


14 


11 



Table includes only those schools whose districts select the instructional materials or provide a list of materials from which schools may choose (Qll). 
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SIE District Curriculum Alignment Questionnaire - Fall 2007 



Q13 



Who was involved in the selection of these 
materials? 


Total 


No 


Yes 


Valid N 


Missing N 


Percent 


Percent 


ql3a: Superintendent 


36 


0 


75 


25 


ql3b: Assistant superintendent 


36 


0 


67 


33 


ql3c: District science specialist/supervisor 


36 


0 


69 


31 


ql3d: Elementary school administrators 


36 


0 


22 


78 


ql3e: Elementary school teachers 


36 


0 


11 


89 


ql3f: Middle school science teachers 


36 


0 


69 


31 


ql3g: High school science teachers 


36 


0 


81 


19 


ql3h: University science educators 


36 


0 


92 


8 


ql3i: University/business/industry scientists 


36 


0 


97 


3 


ql3j: Other (please specify): 


36 


0 


81 


19 



Table includes only those schools whose districts select the instructional materials or provide a list of materials from which schools may choose (Q1 1). 
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SIE District Curriculum Alignment Questionnaire - Fall 2007 



Q15 





Total 


None 


6 or fewer 
hours 


Between 7 
and 12 
hours 


Between 13 
and 18 
hours 


Between 19 
and 24 
hours 


25 or more 
hours 


Valid N 


Missing N 


Percent 


Percent 


Percent 


Percent 


Percent 


Percent 


ql5: About how much professional development did a 
typical teacher receive on these instructional 
materials? 


48 


0 


31 


44 


15 


6 


4 


0 



Q16 



Who provided the professional development? 


Total 


No 


Yes 


Valid N 


Missing N 


Percent 


Percent 


ql6a: The publisher 


33 


0 


42 


58 


ql6b: District science specialist/supervisor 


33 


0 


88 


12 


ql6c: Other district personnel (please 
specify): 


33 


0 


85 


15 


ql6d: ASSET, Inc. 


33 


0 


73 


27 


ql6e: Other (please specify): 


33 


0 


76 


24 



Table includes only those schools where a typical teacher received some professional development on the instructional materials (Q15). 
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Appendix B 



Selected Cohort 2 Teacher Curriculum Survey Results 



Horizon Research, Inc. 
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Grade K-4 Teachers: 

4th Grade Science and Technology Standards 




Figure B-1 
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Appendix C 



Composite Definitions 



Horizon Research, Inc. 



October 2008 




To facilitate the reporting of large amounts of survey data, and because individual questionnaire 
items are potentially unreliable, groups of survey questions that measure similar ideas can be 
combined into “composites.” Each composite represents an important construct related to 
science teaching or professional development. Cronbach’s Coefficient Alpha is a measure of the 
reliability of a composite (i.e., the extent to which the items appear to be measuring the same 
construct). A Cronbach’s Alpha of 0.6 is considered acceptable, 0.7 fair, 0.8 good, and 0.9 
excellent. 

Each composite is calculated by summing the responses to the items associated with that 
composite and then dividing by the total points possible. In order for the composites to be on a 
100-point scale, the lowest response option on each scale was set to 0. As a result, someone who 
marks the lowest point on every item in a composite receives a score of 0, and someone who 
marks the highest point on every item receives a score of 100. It also assures that 50 is the true 
mid-point. The denominator for each composite is determined by computing the maximum 
possible sum of responses for a series of items and dividing by 100; e.g., a nine-item composite 
where each item is on a scale of 0-4 would have a denominator of 0.36. 



Table C-1 

Composite; Teacher Perceptions of Pedagogical Preparedness 



Preparedness to: 


Post-PD 

(Prior) 


Post-PD 

(Now) 


End-of- 

Year 


Use the inquiry-based teaching strategies embedded in the SIE module 


Q7a-p 


Q7a-n 


Q7a 


Use science notebooks to support student learning of the content in the SIE 
module 


Q7b-p 


Q7b-n 


Q7b 


Use the EERA (Eocus, Explore, Reflect, Apply) and 5 E (Engagement, 
Exploration, Explanation, Elaboration, Evaluation) Learning Cycles to 
teach using the SIE module 


Q7c-p 


Q7c-n 


Q7c 


Manage the logistics of the SIE module 


Q7d-p 


Q7d-n 


Q7d 


Handle classroom management issues with the SIE module 


Q7e-p 


Q7e-n 


Q7e 


Use questioning strategies to elicit student thinking about the science concepts 
in the module. 


Q7f-p 


Q7f-n 


Q7f 


Examine student work to assess student thinking about the science concept in 
the module. 


Q7g-p 


Q7g-n 


Q7g 


Teach the science concepts addressed in the module. 


Q7h-p 


Q7h-n 


Q7h 










Number of Items in Composite 


8 


8 


8 


Reliability (Cronbacb’s Coefficient Alpha) 


0.96 


0.93 


0.92 



Horizon Research, Inc 



October 2008 




Table C-2 

Composite: Teacher Perceptions of Pedagogical Content Knowledge 



Understanding of: 


Post-PD 

(Prior) 


Post-PD 

(Now) 


End-of- 

Year 


Student learning goals (big ideas) in the SIE module 


Q6a-p 


Q6a-n 


Q8a 


Science content in the SIE module at the level that the students are 
expected to learn it 


Q6b-p 


Q6b-n 


Q8b 


Science content in the SIE module at a deeper level than what students are 
expected to learn 


Q6c-p 


Q6c-n 


Q8c 


Ideas (either correct or incorrect) that students are likely to have about the 
content in the SIE module before instruction 


Q6d-p 


Q6d-n 


Q8d 


How the activities in the module connect conceptually with one another. 


Q6e-p 


Q6e-n 


Q8e 


How the activities in the module contribute to understanding the big ideas 
of the module. 


Q6f-p 


Q6f-n 


Q8f 


Real-world connections to the science content in the module. 


Q6g-p 


Q6g-n 


Q8g 










Number of Items in Composite 


7 


7 


7 


Reliability (Cronbacb’s Coefficient Alpha) 


0.96 


0.92 


0.93 
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October 2008 




